Aiimi Insight Engine
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Search Flows
      • Search Flow Types
      • Smart Filtering
      • Query and Prompt Classification
      • Search Algorithms
      • Extractive and Generative Models
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • User Security
      • Data and Document Security
      • Source System Security
      • Firewalling
      • Agent Servers
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elastic and Kibana Install (Windows)
    • Aiimi Insight Engine Installation (Windows)
      • Installation Security
      • Certificates in a Key Vault
      • SAR Configuration
      • CSOM Bridge Set Up
      • AI Studio
    • AI Services
      • Prerequisites
      • AI Enrichment Service
        • Installation and Setup
        • Enabling Enrichment Steps
        • Using AI Enrichment Steps
        • Performance and Concurrency
      • AI Model Service
        • Installation and Setup
        • Enabling Providers
        • Private Generative AI
        • Azure Open AI
        • Enabling AI History
        • HTML Cleaner Service
      • Configuration of Logging
      • Offline Set-up of Models
      • Using SSL
      • Running as a Service (Windows)
      • Using GPUs
      • AI and Semantic Search Set Up
        • Open & Closed Book AI
        • Semantic Search
          • Vectors for Semantic Search
          • Source Configuration
          • Sentence Transformer Models
          • Enrichment
          • Kibana
          • Final Search Flow
    • Email Threading Upgrade
  • Run Books
    • SharePoint Online Connector
  • Control Hub
    • Configurations
      • Config Management
      • Security Configurations
        • Security - General
        • Security - Source
          • Active Directory
          • Atlassian
          • Azure Active Directory
          • Builtin Security
          • Miro Security
          • Google Directory
          • Slack Security
        • Security - Sync
        • Security - Agents
        • Security - Scheduling
      • Source Configurations
        • Source - General
        • Source - Source
          • Alfresco Kafka
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • DocuSign
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • Google Drive
          • Google Vault
          • Jira
          • JSON Data Loader
          • Livelink
          • MFiles
          • Microsoft Teams
          • Mimecast
          • Miro
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
            • Azure Portal and Azure AD Authentication
            • Sensitivity Labels
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Slack
          • Versioned Document Store
          • Websites
          • XML Data Loader
        • Source - Crawl
        • Source - Agents
        • Source - Schedule
        • Source - Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • AccessMiner
            • AI Classification
            • Anonymiser
            • CAD Extractor
            • Checksum
            • Content Retrieval
            • Copy
            • Data Rule Processor
            • Delete
            • Email Extractor
            • Entity Rule Processor
            • External Links
            • Geotag
            • Google NLP Extractor
            • Google Vision Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • OcrRest
            • Office Metadata
            • PCI Extractor
            • REST
            • Set Document Risk
            • Text Cleaner
            • Tika Text Extraction
            • Trie Entity Extractor
            • Update Metadata
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • File Extractor
          • GoogleVaultSAR
          • Google Drive Last Access Date
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
    • Credentials
    • Mappings
      • Entities
        • Manage Entity Groups
        • Create an Entity
        • Manage an Entity
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Featured Links
    • AI Settings
      • Classifications
      • Class
      • Class Rules
      • AI Classification
    • User Settings
    • Stats
      • Data Views
    • Global Settings
      • General
      • Authentication
      • App Settings
      • Application Access
      • Thumbnails
      • Presets
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • SAR Disclosure Document Storage
        • Getting SAR data from Google Vault
        • SAR Access
        • SAR File Status
      • Disclosure Portal
        • Disclosure Portal Set Up
        • SARs From The Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Collections
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Timeline Lens Activity Chart
        • Relationship Map
      • Notifications
      • Map Lens
      • Theming
      • User Avatar
      • OData API
      • Uploads
      • Security
    • Search Settings
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Bulk Search
        • Managing a Bulk Search
      • Search Flows
        • Create a Search Flow
          • General
          • Query Classification Step
          • Search Steps
          • Model Steps
      • Filtering
      • Search Performance
      • Related Results
  • AI Studio
    • Classifications
      • Classifications
      • Classification Rules
    • Jobs
  • Labels
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • Whitepapers and Explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • Introduction
  • Input Files
  • Core Properties
  • Model Data Attributes
  • Validate
  1. Control Hub
  2. Configurations
  3. Source Configurations
  4. Source - Source

XML Data Loader

Introduction

If you want to load data structured or unstructured from a XML file use the XML data loader.

To complete this loader you must have a completed Data Model created to match this data. The Data Model must be published to show in the creation of this Data Loader.

Input Files

In order to find the files to load you need to complete the Input Files tab.

  1. Enter the location of the file into the Folder Path.

  2. Enter a naming pattern into file pattern.

    • You can use a wildcard as part of a naming pattern to load multiple files.

    • Entering *.* will load all XML files from that Folder path.

  3. When a file has finished loading you can choose what folder the completed file goes to. Add the name within Complete Folder Path.

    • By default this is set to done.

  4. When processing a file if an error occurs you can choose what folder the file goes to. Add the name within Error Folder Path.

    • By default this is set to error.

    • This allows you to find the files that have failed and investigate why.

  5. You can run a limited number of rows as a test for processing. It will run the process on that number of rows and they will be processed to Insight Engine. Enter the number of rows that should be processed to Rows To Process.

    • If left blank all rows will be processed.

  6. You can choose between XML or XML Line files in the File Format field.

  7. Enter the Object Path using XPath to pinpoint a specific attribute of a file to scan.

  8. If a file is using something other than Pipes to separate multi-values you can choose it from Multi-value Delimiter.

    • The file will not process unless the delimiter is set correctly.

Core Properties

For each file property you can choose its type. By default all properties will be undefined, you can choose between: Undefined, Value, Composite, Constant and Calculated. Fields marked with a red asterisk * require a definition.

  1. Select the property you want to define.

  2. Choose the property type from the list on the right.

    • Depending on the property type selected you will need to define more.

Property Type
Definition

Undefined

These fields will not be mapped.

Value

Select the Column to map this field to.

Composite

Create a string using text and numbers within curly brackets {}. Once you have added the components of the string you can select the inputs that will sit in place of the numbers once processed.

Constant

This will populate a field with the same text value no matter what.

Calculated

This will enter either the current date or Current date and time into a field.

This is the time of the load running.

Model Data Attributes

To connect a field from the file to an attribute in the data model you need to run a file with 3-5 data rows in to the validation.

This will pull in all the field labels from the file. Once run you can then complete the mappings the same way as with core properties.

  1. Select the relevant Business Model.

  2. Select the property you want to define.

  3. Choose the property type from the list on the right.

    • Depending on the property type selected you will need to define more.

Property Type
Definition

Undefined

These fields will not be mapped.

Value

Enter the Column name to map this field to.

Composite

Create a string using text and numbers within curly brackets {}. Once you have added the components of the string you can select the inputs that will sit in place of the numbers once processed.

Constant

This will populate a field with the same text value no matter what.

Calculated

This will enter either the current date or Current date and time into a field.

This is the time of the load running.

Validate

Once all features are defined you can run the test file once more. You can then check the fields and data are mapped correctly. If they are not go back to Model Data Attributes or Core Properties to correct them.

  1. Drag and drop a file for validation or select Upload Here to find a search for a file.

  2. Once loaded select validate.

    • This can be run multiple times until you are happy with the selections.

  3. Select Save when you are done.

PreviousWebsitesNextSource - Crawl