Skip to main content
Connect OpenSearch to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem. The requirements are as follows.
  • For the Unstructured UI or the Unstructured API, local OpenSearch instances are not supported.
  • For Unstructured Ingest, local and non-local OpenSearch instances are supported. For example, to set up an AWS OpenSearch Service instance, complete steps similar to the following:
    1. Sign in to your AWS account, and then open your AWS Management Console.
    2. Open your Amazon OpenSearch Service console.
    3. On the sidebar, expand Managed clusters, and then click Dashboard.
    4. Click Create domain.
    5. In the Name tile, for Domain name, enter some unique domain name for your new OpenSearch instance.
    6. In the Domain creation method tile, select a method for creating the domain. For faster setup with sensible default settings, this example uses the Easy create method. Learn more about the Standard create method.
    7. In the Engine options tile, for Version, AWS recommends that you select the latest version.
    8. In the Network tile, for Network, select a network access method. For faster setup, this example uses the Public access method. Learn more about the VPC access method.
    9. For IP address type, select Dual-stack mode.
    10. In the Fine-grained access control tile, do one of the following:
      • If you want to use an existing AWS IAM user in the AWS account as the domain’s master user, then for Master user, select Set IAM ARN as master user. Then enter the IAM ARN for the master user in the IAM ARN box.
      • If you want to create a master user and password as the domain’s master user instead, then for Master user, select Create master user. Then specify some username and password for this new master user by filling in the Master username, Master password, and Confirm master password fields. Make sure to save the master user’s password in a secure location.
    11. Click Create.
    12. After the domain is created, you must allow Unstructured to access the domain, as follows: a. If the new domain’s settings page is not already showing, open it as follows: in your Amazon Open Search Service console, on the sidebar, expand Managed clusters, and then click Domains. Then, in the list of available domains, click the name of the newly created domain.
      b. On the Security configuration tab, click Edit.
      c. In the Access policy tile, for Domain access policy, select Only use fine-grained access control.
      d. Click Clear policy.
      e. Click Save changes.
    The following video shows how to set up a local OpenSearch instance.
  • The instance’s host identifier (and port number, if you’re using a local OpenSearch instance), as follows:
    • For an AWS OpenSearch Service instance, do the following:
      1. Sign in to your AWS account, and then open your AWS Management Console.
      2. Open your Amazon OpenSearch Service console.
      3. On the sidebar, expand Managed clusters, and then click Dashboard.
      4. In the list of available domains, click the name of your domain.
      5. In the General information tile, copy the value of Domain endpoint v2 (dual stack).
    • For a local OpenSearch instance, see Communicate with OpenSearch.
  • The name of the search index on the instance. For the destination connector, if you need to create an index and you’re using a master user and password as the domain’s master user, you can use for example the following curl command. Replace the following placeholders:
    • Replace <host> with the instance’s host identifier.
    • If you’re using a local OpenSearch instance, replace <port> with the instance’s port number.
    • Replace <master-username> with the master user’s name, and replace <master-password> with the master user’s password.
    • Replace <index-name> with the name of the new search index on the instance.
    • Replace <index-schema> with the schema for the new search index on the instance. A schema is optional; see the explanation following this curl command for more information.
    curl --request PUT "<host>[:<port>]/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
    If you’re using an existing AWS IAM user as the domain’s master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the precedingcurl command. To learn how, see create-index in the AWS CLI Command Reference. For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema, Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless, to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured’s schema. Unstructured cannot provide a schema that is guaranteed to work in all circumstances. This is because these schemas will vary based on your source files’ types; how you want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors. You can adapt the following index schema example for your own needs: 
    {
    "settings": {
        "index": {
            "knn": true,
            "knn.algo_param.ef_search": 100
        }
    },
    "mappings": {
        "properties": {
            "record_id": {
                "type": "text"
            },
            "element_id": {
                "type": "keyword"
            },
            "text": {
                "type": "text"
            },
            "embeddings": {
                "type": "knn_vector",
                "dimension": 384
            },
            "metadata": {
                "type": "object",
                "properties": {
                    "parent_id": {
                        "type": "text"
                    },
                    "page_number": {
                        "type": "integer"
                    },
                    "is_continuation": {
                       "type": "boolean"
                    },
                    "orig_elements": {
                       "type": "text"
                    },
                    "partitioner_type": {
                       "type": "text"
                    }
                }
            }
        }
    }
}
    See also:
  • For non-local OpenSearch instances, or if you’re using basic authentication to a local OpenSearch instance, the master user’s name and password.
  • For local OpenSearch instances, if you’re using certificates for authentication instead of basic authentication:
    • The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
    • The path to the combined private key and certificate file, or
    • The paths to the separate private key and certificate file.
    To learn more, see Authentication backends, HTTP basic authentication, and Client certificate authentication.
The OpenSearch connector dependencies:
CLI, Python
pip install "unstructured-ingest[opensearch]"
You might also need to install additional dependencies, depending on your needs. Learn more. The following environment variables:
  • OPENSEARCH_HOST - The hostname and port number, defined as <hostname>:<port-number> and represented by --hosts (CLI) or hosts (Python).
  • OPENSEARCH_INDEX_NAME - The name of the search index, represented by --index-name (CLI) or index_name (Python).
If you’re using basic authentication to the instance:
  • OPENSEARCH_USERNAME - The user’s name, represented by --username (CLI) or username (Python).
  • OPENSEARCH_PASSWORD - The user’s password, represented by --password (CLI) or password (Python).
If you’re using certificates for authentication instead:
  • OPENSEARCH_CA_CERTS - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by --ca-certs (CLI) or ca_certs (Python).
  • OPENSEARCH_CLIENT_CERT - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by --client-cert (CLI) or client_cert (Python).
  • OPENSEARCH_CLIENT_KEY - The path to the private key file, if OPENSEARCH_CLIENT_CERT refers to just the certificate file. This is represented by --client-key (CLI) or client_key (Python).
Additional related settings include:
  • --use-ssl (CLI) or use_ssl=True (Python) to use SSL for the connection.
  • --verify-certs (CLI) or verify_certs=True (Python) to verify SSL certificates.
  • --ssl-show-warn (CLI) or ssl_show_warn=True (Python) to show a warning when verifying SSL certificates is disabled.
Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector: This example sends data to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page. 
#!/usr/bin/env bash

unstructured-ingest \
  opensearch \
    --index-name $OPENSEARCH_INDEX_NAME \
    --fields "director,plot" \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --hosts $OPENSEARCH_HOST \
    --username $OPENSEARCH_USERNAME \
    --password $OPENSEARCH_PASSWORD \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --strategy hi_res \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the --partition-by-api option (CLI) or partition_by_api (Python) parameter to specify where files are processed:
  • To do local file processing, omit --partition-by-api (CLI) or partition_by_api (Python), or explicitly specify partition_by_api=False (Python). Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
    • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
    • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
    • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL
  • To send files to the Unstructured Partition Endpoint for processing, specify --partition-by-api (CLI) or partition_by_api=True (Python). Unstructured also requires an Unstructured API key and API URL, by adding the following:
    • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
    • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
    • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL, representing your API key and API URL, respectively.
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to Let’s Go, Pay-As-You-Go, and Business SaaS accounts.The default API URL for Unstructured Ingest is https://api.unstructuredapp.io/general/v0/general, which is the API URL for the Unstructured Partition Endpoint. However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at support@unstructured.io.If you do not have an API key, get one now.If you are using a Business account, the process for generating Unstructured API keys, and the Unstructured API URL that you use, are different. For instructions, see your Unstructured account administrator, or email Unstructured Support at support@unstructured.io.