Skip to main content

Collections

After artemis has completed enrollment it will start to poll the server for collection jobs. These collection jobs will tell what artifacts artemis should parse and upload the server. The artemis collection process has two workflows:

  • Request collection jobs
  • Upload collection results

Collection Request​

The artemis daemon will send a POST request to the collection URI defined in the server.toml file.

For example, the server.toml file below:

[server]
url = "http://127.0.0.1"
port = 8000
ignore_ssl = false
enrollment = "endpoint/enroll"
collections = "endpoint/collections"
config = "endpoint/config"
version = 1
key = "my key"

the daemon will send a collection request to the URI /v1/endpoint/collections.

Collection Request Payload​

The artemis daemon will send the following JSON payload when requesting collection jobs.

{
    "endpoint_id": "endpoint id generated by the server"
}

An example payload is below:

{
    "endpoint_id": "88c4f69e-9953-4530-9bfc-8a71afe9f18e"
}

Collection Request Response​

The artemis daemon expects a response continaing the following:

{
    "collection":"base64 TOML collection file",
    "endpoint_invalid":"boolean value. False means the daemon should re-enroll"
}

The TOML file should have the same format as standard collection TOML file and should have the following fields:

  • name: Collection name
  • endpoint_id: ID generated by the server. The daemon receives this ID upon enrollment.
  • collection_id: A number assigned to the collection. This can be any postive number
  • directory: Directory artemis should store the results if it cannot upload to the server. If provided this will override the configuration collection path.
  • output: Output type. This can be LOCAL, GCP, AZURE, AWS, or API.
  • format: Format for the output. Can be JSON, CSV, or JSONL. If you use the API output option the format will always be JSONL.
  • compress: If the data should be compressed with gzip. If enabled all uploads to GCP, AZURE, AWS, or API will be gzip compressed. If the output is LOCAL the data will compressed with gzip and then the output directory will compressed with zip. If you use the API output option the data will always be gzip compressed.
  • timeline: Whether the data should be timelined locally before outputted.
  • url: The URL associated with either aws, gcp, or azure. This is required if using GCP, AWS, AZURE output.
    • api_key: The API key associated with either AWS, AZURE, GCP. This is required only if using GCP, AWS, AZURE output.
  • filter_name: The name of the provided filter_script. This is optional but if you are using a filter_script you should provide a name. Otherwise the default name UnknownFilterName is used
  • filter_script: An advanced optional field. Artemis will pass the results of each [[artifacts]] entry into a script. See scripting section for detailed overview of this option.
  • logging: Set the logging level for artemis. This is optional by default artemis will log errors and warnings. Valid options are: warn, error, debug, or info
  • [[artifacts]] A list of artifacts to collect
    • artifact_name: Name of artifact
    • filter: Whether to filter the artifact data through the filter_script. This is optional by default nothing is filtered. This option will send artifact results to your provided filter_script
    • [artifacts.NAME] Artifact configuration parameters
      • artifact options: Options for the artifact you want to collect

Collection Upload Request​

If you are using the API option as your output type the artemis daemon will attempt to upload the results to your remote server.
The artemis daemon will send a POST request to the collection URI defined in the server.toml file.

For example, the server.toml file below:

[server]
url = "http://127.0.0.1"
port = 8000
ignore_ssl = false
enrollment = "endpoint/enroll"
collections = "endpoint/collections"
config = "endpoint/config"
version = 1
key = "my key"

the daemon will send a collection upload request to the URI /v1/endpoint/collections/upload.

Collection Upload Payload​

The artemis daemon will upload data to the collection URI defined in the server.toml file. If you choose the API output option the daemon will send gzip compressed POST requests to the remote server containing the parsed forensic data.

The daemon will added the following headers to requests:

  • x-artemis-endpoint_id: The ID assigned to the endpoint. This ID was generated by the server upon enrollment
  • x-artemis-collection_id: The collection ID. This is taken from the collection TOML file
  • x-artemis-collection_name: The collection name. This is taken from the collection TOML file

Uploaded gzip data will have the Content-Encoding header set to gzip and the MIME type application/jsonl.

Uploads are multipart uploads and will have a random UUID assigned as the filename.

When the collection is completed, artemis will send two final requests to signify the end of the collection:

  • Artemis will upload a <uuid>.log file that contains artemis logging information related to the collection.
  • Artemis will upload a status.log file that contains an artifact mapping to the UUID filenames that were uploaded. The artifact name can also be found in the uploaded JSONL data

The final request will have the header:

  • x-artemis-collection-complete: Boolean value indicating the collection is done and no more uploads associated with the collection ID will happen.

The JSONL output format is the same format when running artemis as a CLI tool