Aiimi Insight Engine
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • User Security
      • Data and Document Security
      • Source System Security
      • Firewalling
      • Agent Servers
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elastic and Kibana Install (Windows)
    • Aiimi Insight Engine Installation (Windows)
      • Installation Security
      • Certificates in a Key Vault
      • SAR Configuration
      • CSOM Bridge Set Up
      • AI Studio
    • AI Services
      • Prerequisites
      • AI Enrichment Service
        • Installation and Setup
        • Enabling Enrichment Steps
        • Using AI Enrichment Steps
        • Performance and Concurrency
      • AI Model Service
        • Installation and Setup
        • Enabling Providers
        • Private Generative AI
        • Azure Open AI
        • Enabling AI History
        • HTML Cleaner Service
      • Configuration of Logging
      • Offline Set-up of Models
      • Using SSL
      • Running as a Service (Windows)
      • Using GPUs
      • AI and Semantic Search Set Up
        • Open & Closed Book AI
        • Semantic Search
          • Vectors for Semantic Search
          • Source Configuration
          • Sentence Transformer Models
          • Enrichment
          • Kibana
          • Final Search Flow
    • Email Threading Upgrade
  • Run Books
    • SharePoint Online Connector
      • Migrating ACS to Azure AD with Sites.Read.All
      • Migrating ACS to Azure AD with Sites.FullControl.All
  • Control Hub
    • Agents
      • Configurations
        • Config Management
        • Security Configurations
          • Security - General
          • Security - Source
            • Active Directory
            • Atlassian
            • Azure Active Directory
            • Builtin Security
            • Miro Security
            • Google Directory
            • SharePoint Security
            • Slack Security
          • Security - Sync
          • Security - Agents
          • Security - Scheduling
        • Source Configurations
          • Source - General
          • Source - Source
            • Alfresco Kafka
            • Azure Blob Storage
            • BBC Monitoring
            • Big Query Cataloguer
            • BIM360
            • CSV Data Loader
            • Confluence
            • Content Server
            • Data File Cataloguer
            • Document Store
            • DocuSign
            • Dropbox
            • Exchange 365
            • Filesystem
            • Google Bucket
            • Google Drive
            • Google Vault
            • Jira
            • JSON Data Loader
            • Livelink
            • Microsoft Teams
            • Mimecast
            • Miro
            • ODBC Data Loader
            • PowerBi Cataloguer
            • Reuters Connect
            • ShareFile
            • SharePoint
              • Azure Portal and Azure AD Authentication
              • Sensitivity Labels
            • SharePoint Legacy
            • SQL Server Cataloguer
            • Slack
            • Versioned Document Store
            • Websites
            • XML Data Loader
          • Source - Crawl
          • Source - Agents
          • Source - Schedule
          • Source - Advanced
        • Enrichment Configurations
          • Creating a Pipeline
            • General
            • Steps
              • AccessMiner
              • AI Classification
              • Apply Sensitivity Label
              • Anonymiser
              • CAD Extractor
              • Checksum
              • Content Retrieval
              • Copy
              • Data Rule Processor
              • Delete
              • Direct Copy
              • Email Extractor
              • Entity Rule Processor
              • External Links
              • Geotag
              • Google NLP Extractor
              • Google Vision Extractor
              • Metrics Calculation
              • Microsoft Vision Extractor
              • OcrRest
              • Office Metadata
              • PCI Extractor
              • REST
              • Set Document Risk
              • Text Cleaner
              • Tika Text Extraction
              • Trie Entity Extractor
              • Update Metadata
            • Filters
            • Agents
            • Schedule
            • Advanced
        • OCR Engine
        • Job Configurations
          • General
          • Job
            • AutomatedSearchJob
            • Command Job
            • ElasticJob
            • Extended Metrics Job
            • File Extractor
            • GoogleVaultSAR
            • Google Drive Last Access Date
            • Nightly Events Processor Job
            • Notifications Processor Job
            • Portal Sync Job
            • Purge Job
            • SAR Archiving
            • Text Content Merge Job
          • Output
          • Agents
          • Scheduling
        • Migration Configuration
          • General
          • Filter
          • Metadata Mappings
          • Agents
          • Scheduling
          • Advanced
      • Stats
        • Data Views
    • Security
      • User Settings
      • Credentials
      • Authentication
      • Application Access
      • Auditing
      • Descriptor Groups
      • Uploads
    • Mappings
      • Entities
        • Manage Entity Groups
        • Create an Entity
        • Manage an Entity
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Search Settings
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Bulk Search
        • Managing a Bulk Search
      • Filtering
      • Search Performance
      • Related Results
      • Featured Links
    • AI Settings
      • Search Flows
        • Search Flow Types
        • General Configuration
        • Query and Prompt Classification
        • Search Steps
        • Smart Filtering
        • Model Steps
        • Result Templates
        • System Prompt
      • Tools
        • Concepts
        • Import OOTB Tools
        • Built In Functions and Tools
        • Create and Edit Tools
      • Classifications
        • Class
        • Class Rules
        • AI Classification
    • User Interface
      • Thumbnails
      • Code of Conduct
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Timeline Lens Activity Chart
        • Relationship Map
      • Map Lens
      • Theming
      • User Avatar
    • Global Settings
      • General
      • App Settings
      • Presets
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • SAR Disclosure Document Storage
        • Getting SAR data from Google Vault
        • SAR Configuration Access
        • SAR File Status
      • Disclosure Portal
        • Disclosure Portal Set Up
        • SARs From The Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Collections
      • Notifications
      • OData API
  • AI Studio
    • Classifications
      • Classifications
      • Classification Rules
    • Jobs
  • Labels
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Library
      • Data Science API Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
      • Search API Wrapper
        • Login
        • Privileged Access
        • Search
        • Collection
        • ChatBot Class
      • Admin API Wrapper
      • AI Model Server API Wrapper
      • Utilities
        • Query Builders
        • Azure Key Vault Wrapper
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • Whitepapers and Explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  1. API Guides
  2. Data Science API Guide
  3. Utilities

Query Builders

PreviousUtilitiesNextAzure Key Vault Wrapper

Last updated 3 months ago

CtrlK
  • SubQuery
  • BoolQuery
  • FieldQuery
  • ExistsQuery
  • RegexQuery
  • WildcardQuery
  • PrefixQuery
  • RangeQuery
  • FuzzyQuery
  • QueryString

One way to write InsightMaker queries is using the query builder helper objects available as part of the library. All query builder objects support the python and (&), or (|) and not (~) logical operators allowing for plenty of flexibility.

SubQuery

SubQuery(method, content) All helpers inherit from the base class, SubQuery. It has two key properties, method and content, corresponding to a key, value pair, i.e. {method: content}. It implements the following functions:

SubQuery.query_json() The most important method of SubQuery, this returns the query wrapped in a Bool Query, suitable for use with the DS API.

SubQuery.json() The query without a bool wrapper.

SubQuery.content_json() The query json for the content element of this SubQuery only.

SubQuery.append(other) Append another SubQuery of equivalent method to the content of this SubQuery. (This exists primarily to assist the logical operator overload functions)

SubQuery.merge(new_content) Append additional content to the content of this SubQuery. (This exists primarily to assist the logical operator overload functions)

SubQuery.is_bool_occurence() Is this SubQuery of method “must”, “filter”, “should” or “must_not”, all of which must exist within a Bool Query. (This exists primarily to assist the logical operator overload functions)


BoolQuery

BoolQuery(occurrences, minimum_should_match=1) A way to logically combine other SubQuery objects. You shouldn't need to create these as using the logical operators and (&), or (|) and not (~) on SubQuery objects will automatically create them for you. Elastic Query DSL reference.

Type
Description

occurrences

list

A list of SubQuery objects of method: “must”, “filter”, “should” or “must_not”

minimum_should_match

numeric

The number or percentage of should clauses returned documents must match. Default: 1


FieldQuery

FieldQuery(field, values) An Elastic Query DSL Terms Query, it returns documents where a given field matches certain values.

Name
Type
Description

field

string

The name of the field to match on

values

list

A list of values (of appropriate type) to match the field on


ExistsQuery

ExistsQuery(field) Filters to documents where a non-null/empty value exists for the given field. Reference.

Name
Type
Description

field

string

The name of the field to check


RegexQuery

RegexQuery(field, expression, flags=None, max_determined_states=None) An Elastic Query DSL Regular Expression Query.

Name
Type
Description

field

string

The name of the field to check

expression

string

The regular expression string

flags

string

Optional - | separated flags of ALL (default), ANYSTRING, COMPLEMENT, EMPTY, INTERSECTION, INTERVAL, or NONE. As defined by Lucene.

max_determined_states

integer

Optional – A way of limiting the number of execution states for regex matches, useful for more complex regular expressions. Elasticsearch default: 10,000.


WildcardQuery

WildcardQuery(field, wildcard) An Elastic Query DSL Wildcard Query. Slightly simpler than a full blown regular expression query, just using * to match zero or more characters.

Name
Type
Description

field

string

The name of the field to check

wildcard

string

The wildcard string to use


PrefixQuery

PrefixQuery(field, prefix) An Elastic Query DSL Prefix Query. Used to match documents where a field starts with a given string.

Name
Type
Description

field

string

The name of the field to check

prefix

string

The prefix string to use


RangeQuery

RangeQuery(field, min_value, max_value, lower_inclusive=True, upper_inclusive=False) An Elastic Query DSL Range Query.

Name
Type
Description

field

string

The name of the field to check

min_value

numeric

The lower bound to match above

max_value

numeric

The upper bound to match below

lower_inclusive

boolean

Include exact matches on the lower bound. Default: True

upper_inclusive

boolean

Include exact matches on the upper bound. Default: False


FuzzyQuery

FuzzyQuery(field, search_t, fuzziness=None, prefix_length=None, max_expansions=None, transpositions=None) An Elastic Query DSL Fuzzy Query.

Name
Type
Description

field

string

The name of the field to check

search_t

string

The fuzzy search string

fuzziness

integer

The edit distance to match from. See documentation. Defaults to automatic.

prefix_length

integer

The number of initial characters to not consider part of the fuzzy match. Default: 0

max_expansions

integer

The maximum number of terms that the fuzzy query will expand to. Defaults to 50.

transpositions

boolean

If character transpositions are supported in the match (i.e. ab->ba would have a distance of 1 rather than 2). Default: False


QueryString

QueryString(query, params=None) An Elasticsearch default string query. Supports Lucene syntax.

Name
Type
Description

query

string

The query string

params

dictionary

Optional – Additional parameters as defined by the Elasticsearch documentation.