task create

Description

Create a batch of tasks from files in a local directory or files in Summit's S3 Bucket.

Usage

summit task create {S3_URI|LOCAL_PATH} [flags]

LOCAL_PATH represents the path of a local folder.
S3_URI represents the location of a Summit S3 folder. This must be written in the form s3://summit-client-assets-ie/myclientid/myfolder where myfolder can have additional subfolders

Flags

    --upload-path-prefix string   upload assets to S3, but to the specified path prefix(folder)
    --output string               output task data to a .csv or .json file instead of sending a task creation request
    --from-file                   use local .csv or .json file to create tasks
    --monitor                     display and monitor the status of the batch immediately after running this command
    --in-batches                  create batches based on each subfolder
    --in-client-batches           create batches based on each subfolder, and assign the Client Batch ID using the subfolder's name
    --client-batch-id string      specify the Client Batch ID string
    --group-size int              automatically assign Task Group IDs to tasks based on a specified group size
    --modified-after string       only include assets that were modified after given datetime string
    --modified-before string      only include assets that were modified before given datetime string
    --ext string                  filter files based on comma separated list of file extensions
    --deep                        (deprecated)include files deeply nested in subfolders
-h, --help                        help for create

Examples

Creating a batch of tasks given a local path

The mapping json can be automatically be generated during init. e.g.

    {
        "name": "{{ .BaseName }}",
        "url": "{{.URL}}"
    }

The following 'task create' command creates a task for each video file. Assume the following asset folder exists locally:

    assets/
        assets_batch001/
            video001.mp4
            video002.mp4
            video003.mp4

Command

$ summit task create assets/assets_batch001/

Result

A task will be created for video001.mp4

    "name" : video001.mp4
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/video001.mp4"

A task will be created for video002.mp4

    "name" : video002.mp4
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/video002.mp4"

A task will be created for video003.mp4

    "name" : video003.mp4
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/video003.mp4"

Creating a batch of tasks given a S3 path

The mapping json can be automatically be generated during init. e.g.

    {
        "name": "{{ .BaseName }}",
        "url": "{{.URL}}"
    }

The following 'task create' command creates a task for each video file. Assume the following asset folder already exists in S3:

    s3://summit-client-assets-ie/123/assets/assets_batch002/
        video001.mp4
        video002.mp4
        video003.mp4

Command

$ summit task create s3://summit-client-assets-ie/123/assets/assets_batch002/

Result

A task will be created for video001.mp4:

    "name" : video001.mp4
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch002/video001.mp4"

A task will be created for video002.mp4:

    "name" : video002.mp4
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch002/video002.mp4"

A task will be created for video003.mp4:

    "name" : video003.mp4
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch002/video003.mp4"

Creating a batch of tasks that contains multiple URL inputs; for example, a 3D project with 2D helper videos

The mapping json will be automatically be generated during init as, for example:

    {
        "name": "{{ .BaseName }}",
        "3d_url_input": "{{.URL}}",
        "3d_sensor_location" : "{{.URL}}",
        "2d_back_url_input": "{{.URL}}",
        "2d_front_url_input": "{{.URL}}"
    }

However, since there are multiple URL inputs, the folder or files containing the correct assets need to be specified.
Folder example (don't forget to include the trailing "/" at the end of the folder names):

    {
        "name": "{{ .BaseName }}",
        "3d_url_input": "{{.URL}}lidar/",
        "3d_sensor_location" : "{{.URL}}sensor_location/",
        "2d_back_url_input": "{{.URL}}back_video/",
        "2d_front_url_input": "{{.URL}}front_video/"
    }

File example:

    {
        "name": "{{ .BaseName }}",
        "3d_url_input": "{{.URL}}lidar.zip",
        "3d_sensor_location" : "{{.URL}}sensor_location.zip",
        "2d_back_url_input": "{{.URL}}back_video.mp4",
        "2d_front_url_input": "{{.URL}}front_video.mp4"
    }

The following 'task create' command creates a task for each folder which is a sequence. Assume the following asset folder exists locally (you would use the mapping json with the folder example above):

    assets/
        assets_batch001/
            sequence1/
                 lidar/
                    001.pcd
                    002.pcd
                 back_video/
                    001.jpg
                    002.jpg
                 front_video/
                    001.jpg
                    002.jpg
                 sensor_location/
                    001.csv
                    002.csv
            sequence2/
                 lidar/
                    001.pcd
                    002.pcd
                 back_video/
                    001.jpg
                    002.jpg
                 front_video/
                    001.jpg
                    002.jpg
                 sensor_location/
                    001.csv
                    002.csv

Command

$ summit task create assets/assets_batch001

Result

A task will be created for sequence1/:

    "name" : sequence1
    "3d_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch001/sequence1/lidar/"
    "3d_sensor_location" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch001/sequence1/sensor_location/"
    "2d_back_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch001/sequence1/back_video/"
    "2d_front_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch001/sequence1/front_video/"

A task will be created for sequence2/:

    "name" : sequence2
    "3d_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/sequence2/lidar/"
    "3d_sensor_location" :"https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/sequence2/sensor_location/"
    "2d_back_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/sequence2/back_video/"
    "2d_front_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/sequence2/front_video/"

Creating a batch of tasks that contains multiple URL inputs; for example, a project with multiple 2D videos

The mapping json will be automatically be generated during init as, for example:

    {
        "name": "{{ .BaseName }}",
        "2d_back_url_input": "{{.URL}}",
        "2d_front_url_input": "{{.URL}}"
    }

    {
        "name": "{{ .BaseName }}",
        "2d_back_url_input": "{{.URL}}back_video/",
        "2d_front_url_input": "{{.URL}}front_video/"
    }

File example:

    {
        "name": "{{ .BaseName }}",
        "2d_back_url_input": "{{.URL}}back_video.mp4",
        "2d_front_url_input": "{{.URL}}front_video.mp4"
    }

    assets/
        assets_batch001/
            sequence1/
                 back_video/
                    001.jpg
                    002.jpg
                 front_video/
                    001.jpg
                    002.jpg
            sequence2/
                 back_video/
                    001.jpg
                    002.jpg
                 front_video/
                    001.jpg
                    002.jpg

Command

$ summit task create assets/assets_batch001

Result

A task will be created for sequence1/:

    "name" : sequence1
    "2d_back_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch001/sequence1/back_video/"
    "2d_front_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets/assets_batch001/sequence1/front_video/"

A task will be created for sequence2/:

    "name" : sequence2
    "2d_back_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/sequence2/back_video/"
    "2d_front_url_input" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/sequence2/front_video/"

Monitoring the status of a batch immediately from task create command

Add the --monitor flag to 'task create'

Command

$ summit task create assets_batch002 --monitor

Result

Batch 113160: imported: 0/2 tasks completed

Creating a batch of tasks and specifying a client batch id

The mapping json can be automatically be generated during init. e.g.

    {
        "url": "{{.URL}}",
        "name": "{{ .BaseName }}",
        "client batch id": "{{ .ClientBatchID }}"
     }

The following 'task create' command creates task for each file. Assume the following assets exists locally:

    assets/
        assets_batch001/
            img001.png
            img002.png
            img003.png

Command

$ summit task create assets/assets_batch001 --client-batch-id "my_batch_001"

Result

A batch will create individual tasks for img001.png, img002.png, img003.png and 'client batch id' is set to 'my_batch_001'.

Creating multiple batch of tasks where each parent folder is considered a separate batch

The mapping file can be automatically be generated during init. e.g.

    {
        "name": "{{ .BaseName }}",
        "url": "{{.URL}}"
    }

The following 'task create' command creates task for each file. Assume the following assets exists locally:

    assets/
        assets_batch001/
            img001.png
            img002.png
            img003.png
        assets_batch002/
            img004.png
            img005.png
            img006.png

Command

$ summit task create assets --in-batches

Result

Batch 1 will create individual tasks for img001.png, img002.png, img003.png.
Batch 2 will create individual tasks for img004.png, img005.png, img006.png.

Creating multiple batch of tasks where each parent folder is considered a separate batch, and automatically assign the client batch id as the parent folder

The mapping file can be automatically be generated during init. e.g.

    {
        "name": "{{ .BaseName }}",
        "url": "{{.URL}}",
        "client batch id": "{{ .ClientBatchID }}"
    }

The following 'task create' command creates task for each file. Assume the following assets exists locally:

    assets/
        assets_batch001/
            img001.png
            img002.png
            img003.png
        assets_batch002/
            img004.png
            img005.png
            img006.png

Command

$ summit task create ./assets --in-client-batches

Result

A batch will create individual tasks for img001.png, img002.png, img003.png, and 'client batch id' is set to assets_batch001.
The same batch will also create img004.png, img005.png, img006.png and 'client batch id' is set to assets_batch002.

Create a batch of tasks and group them according to size

The mapping file can be automatically be generated during init. e.g.

    {
        "example_input_url": "{{.URL}}",
        "name": "{{ .BaseName }}",
        "example_input_batch_id": "{{ .GroupID }}"
     }

The following 'task create' command creates task for each file. Assume the following assets exists locally:

    assets/
        assets_batch001/
            img001.png
            img002.png
            img003.png
            img004.png
            img005.png

Command

$ summit task create assets/assets_batch001 --group-size 2

Result

Invididual tasks are created for img001.png, img002.png and are assigned to the same group id (auto assigned).
Invididual tasks are created for img003.png, img004.png and are assigned to the same group id (auto assigned).
Invididual tasks are created for img005.png and are assigned to the same group id (auto assigned).

Creating a batch of tasks and filter specific file types

The mapping json can be automatically be generated during init. e.g.
    {
        "name": "{{ .BaseName }}",
        "url": "{{.URL}}"
    }

The following 'task create' command creates a task for each video file. Assume the following asset folder exists locally:
    assets/
        assets_batch001/
            img1.jpg
            img2.png
            file2.txt
            unsupported.jp2


$ summit task create assets/assets_batch001/ --ext ".jpg,.png"

Output:
    A task will be created for img1.jpg:
        "name" : img1.jpg
        "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/img1.jpg"
    A task will be created for img2.png:
        "name" : img2.png
        "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/img2.png"

Creating a batch of tasks by finding recursively finding files in target folder (deprecated)

The mapping json can be automatically be generated during init. e.g.

    {
        "name": "{{ .BaseName }}",
        "url": "{{.URL}}"
    }

The following 'task create' command creates a task for each image file. Assume the following asset folder exists locally:

    assets/
        assets_batch001/
            img1.jpg
            subfolder1/
                img2.jpg
                subfolder2/
                    img3.jpg

Command

$ summit task create assets/assets_batch001/ --deep

Output

A task will be created for img1.jpg:

    "name" : img1.jpg
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/img1.jpg"

A task will be created for img2.jpg:

    "name" : img2.jpg
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/subfolder1/img2.jpg"

A task will be created for img3.jpg:

    "name" : img3.jpg
    "url" : "https://summit-client-assets-ie.s3.amazonaws.com/123/assets_batch001/subfolder1/subfolder2/img3.jpg"

Creating an export of a batch of tasks to CSV or JSON file

Command

$ summit task create assets_batch001 --output tasks.csv
$ summit task create assets_batch001 --output tasks.json

Creating a batch of tasks from a CSV or JSON file

Command

$ summit task create --from-file tasks.csv
$ summit task create --from-file tasks.json

Creating a batch of tasks from assets that were modified within a date range

Command

$ summit task create assets/assets_batch001 –modified-after 2021-06-10T00:00:00Z –modified-before 2021-06-24T00:00:00Z