Aiimi Insight Engine Disco
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
      • Development Environments
      • Testing and QA Environments
      • Production Environments
      • Disaster Recovery Environments
    • Security
      • Source Systems
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elasticsearch Installation (Windows)
    • Aiimi Insight Engine Installation (Windows)
    • Python REST Service Install and Configuration
      • Installation and Setup
        • Offline Huggingface Set-up
        • Configuration of Logging
        • Running as a Service (Windows)
        • Using SSL
        • Performance and Concurrency
        • Security
      • Communicating with the Python REST Service
      • Step Configuration
        • Classify
        • Entity Mapper
        • Huggingface NER
        • Language Detection
        • Phrases and Topics
        • Summary
        • Spacyner
    • HTML Cleaner Service
  • security
    • Users
      • User Authentication
      • Code of Conduct
      • App Access and Role-Based Access
    • Data & Documents
      • Document and Data Access Control Lists
      • Progressive Access and Privileged Access
        • Progressive Access
        • Privileged Access
      • Source Scope and Visibility within Apps
      • Classification
      • Advanced Security Controls - Security Classifications
      • Risk Ratings
      • Redaction and Anonymisation
      • Banned Words
      • Mark as Sensitive
      • Audit Controls
  • Control Hub
    • Configurations
      • Managing Configurations
        • Edit and Delete
        • Find a Configuration
        • Monitoring Stats
        • Stop a Configuration
        • Refreshing
      • Security Configurations
        • Configuring Security
        • Active Directory
        • Azure Active Directory
        • BuiltinSecurity
        • Security Sync
        • Security Agents
        • Security Scheduling
      • Source Configurations
        • General
        • Source
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • JSON Data Loader
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Websites
          • XML Data Loader
        • Crawl
        • Source Agents
        • Source Scheduling
        • Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • Tika Text Extraction
            • External Links
            • Delete
            • Copy
            • Text Cleaner
            • Data Rule Processor
            • Checksum
            • OcrRest
            • AccessMiner
            • CAD Extractor
            • Trie Entity Extractor
            • PCI Extractor
            • Email Extractor
            • Geotag
            • Google Vision Extractor
            • Google NLP Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • Entity Rule Processor
            • Anonymiser
            • Set Document Risk
            • Content Retrieval
            • REST
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
      • Content Server
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Entities
      • Create a Group
      • Create an Entity
      • Managing Entities
      • Import or Export Entities
    • Models
      • Create a New Model
      • Find a Model
      • Enable or Disable a Model
    • Featured Links
    • Global Settings
      • General
        • Thumbnails
        • Windows Authentication
        • SAML2 Authentication
        • ADFS
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Collections
        • Miscellaneous
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
        • Accurate Totals for Pagination
        • Source cards
        • Facets
      • Filtering
      • Presets
        • File Groups
        • Security Groups
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • Getting SAR data from Google Vault
        • Redacting Information
        • SAR Access
      • Privacy Portal
        • Activate the Privacy Portal
        • Disclosure
        • Submit SARs From The Privacy Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Create and Modified Date Activity Chart
        • Relationship Map
      • Notifications
      • Map View
      • App
      • Theming
        • Details
        • Logo
        • Background Images
        • Colours
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
    • User Settings
    • Stats
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • SubQuery
  • BoolQuery
  • FieldQuery
  • ExistsQuery
  • RegexQuery
  • WildcardQuery
  • PrefixQuery
  • RangeQuery
  • FuzzyQuery
  • QueryString
  1. API Guides
  2. Data Science API Guide
  3. Python Wrapper

Query Builders

PreviousSearchNextScroll

The third way of writing InsightMaker queries is by using the query builder helper objects available as part of the library, using these the Elastic DSL query in the previous section can be achieved by:

from insight_maker.api.query_builders import *

query = RangeQuery("size", 1000, 2000) & ~RegexQuery("name", ".*\.txt")

results = im.search(query, dataset = "documents", size = 10)

print(results.df)

All query builder objects support the python and (&), or (|) and not (~) logical operators allowing for plenty of flexibility.

The available helper objects are as follows:

SubQuery

SubQuery(method, content)

All helpers inherit from the base class, SubQuery, it has two key properties, method and content, corresponding to a key, value pair, i.e. {method: content}. It implements the following functions:

  • SubQuery.query_json() The most important method of SubQuery, this returns the query wrapped in a , suitable for use with the DS API.

  • SubQuery.json() The query without a bool wrapper.

  • SubQuery.content_json() The query json for the content element of this SubQuery only.

  • SubQuery.append(other) Append another SubQuery of equivalent method to the content of this SubQuery. (This exists primarily to assist the logical operator overload functions)

  • SubQuery.merge(new_content) Append additional content to the content of this SubQuery. (This exists primarily to assist the logical operator overload functions)

  • SubQuery.is_bool_occurence() Is this SubQuery of method “must”, “filter”, “should” or “must_not”, all of which must exist within a Bool Query. (This exists primarily to assist the logical operator overload functions)

BoolQuery

BoolQuery(occurrences, minimum_should_match=1)

Parameters:

Type
Description

occurrences

list

A list of SubQuery objects of method: “must”, “filter”, “should” or “must_not”

minimum_should_match

numeric

The number or percentage of should clauses returned documents must match. Default: 1

FieldQuery

FieldQuery(field, values)

Parameters:

Name
Type
Description

field

string

The name of the field to match on

values

list

A list of values (of appropriate type) to match the field on

ExistsQuery

ExistsQuery(field)

Parameters:

Name
Type
Description

field

string

The name of the field to check

RegexQuery

RegexQuery(field, expression, flags=None, max_determined_states=None)

Parameters:

Name
Type
Description

field

string

The name of the field to check

expression

string

The regular expression string

flags

string

max_determined_states

integer

Optional – A way of limiting the number of execution states for regex matches, useful for more complex regular expressions. Elasticsearch default: 10,000.

WildcardQuery

WildcardQuery(field, wildcard)

Parameters:

Name
Type
Description

field

string

The name of the field to check

wildcard

string

The wildcard string to use

PrefixQuery

PrefixQuery(field, prefix)

Parameters:

Name
Type
Description

field

string

The name of the field to check

prefix

string

The prefix string to use

RangeQuery

RangeQuery(field, min_value, max_value, lower_inclusive=True, upper_inclusive=False)

Parameters:

Name
Type
Description

field

string

The name of the field to check

min_value

numeric

The lower bound to match above

max_value

numeric

The upper bound to match below

lower_inclusive

boolean

Include exact matches on the lower bound. Default: True

upper_inclusive

boolean

Include exact matches on the upper bound. Default: False

FuzzyQuery

FuzzyQuery(field, search_t, fuzziness=None, prefix_length=None, max_expansions=None, transpositions=None)

Parameters:

Name
Type
Description

field

string

The name of the field to check

search_t

string

The fuzzy search string

fuzziness

integer

prefix_length

integer

The number of initial characters to not consider part of the fuzzy match. Default: 0

max_expansions

integer

The maximum number of terms that the fuzzy query will expand to. Defaults to 50.

transpositions

boolean

If character transpositions are supported in the match (i.e. ab->ba would have a distance of 1 rather than 2). Default: False

QueryString

QueryString(query, params=None)

Parameters:

Name
Type
Description

query

string

The query string

params

dictionary

A way to logically combine other SubQuery objects – typically you should not need to create these, rather using the logical operators and (&), or (|) and not (~) on SubQuery objects will automatically create them for you. .

An Elastic Query DSL , returns documents where a given field matches certain values.

Filters to documents where a non-null/empty value exists for the given field. .

An Elastic Query DSL .

Optional - | separated flags of ALL (default), ANYSTRING, COMPLEMENT, EMPTY, INTERSECTION, INTERVAL, or NONE. As .

An Elastic Query DSL . Slightly simpler than a full blown regular expression query, just using * to match zero or more characters.

An Elastic Query DSL . Used to match documents where a field starts with a given string.

An Elastic Query DSL .

An Elastic Query DSL .

The edit distance to match from. See . Defaults to automatic.

An Elasticsearch default . Supports Lucene syntax.

Optional – Additional parameters as defined by the .

Bool Query
Elastic Query DSL reference
Terms Query
Reference
Regular Expression Query
Wildcard Query
Prefix Query
Range Query
Fuzzy Query
string query
defined by Lucene
documentation
Elasticsearch documentation