Aiimi Insight Engine
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • User Security
      • Data and Document Security
      • Source System Security
      • Firewalling
      • Agent Servers
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elastic and Kibana Install (Windows)
    • Aiimi Insight Engine Installation (Windows)
      • Installation Security
      • Certificates in a Key Vault
      • SAR Configuration
      • CSOM Bridge Set Up
      • AI Studio
    • AI Services
      • Prerequisites
      • AI Enrichment Service
        • Installation and Setup
        • Enabling Enrichment Steps
        • Using AI Enrichment Steps
        • Performance and Concurrency
      • AI Model Service
        • Installation and Setup
        • Enabling Providers
        • Private Generative AI
        • Azure Open AI
        • Enabling AI History
        • HTML Cleaner Service
      • Configuration of Logging
      • Offline Set-up of Models
      • Using SSL
      • Running as a Service (Windows)
      • Using GPUs
      • AI and Semantic Search Set Up
        • Open & Closed Book AI
        • Semantic Search
          • Vectors for Semantic Search
          • Source Configuration
          • Sentence Transformer Models
          • Enrichment
          • Kibana
          • Final Search Flow
    • Email Threading Upgrade
  • Run Books
    • SharePoint Online Connector
      • Migrating ACS to Azure AD with Sites.Read.All
      • Migrating ACS to Azure AD with Sites.FullControl.All
  • Control Hub
    • Agents
      • Configurations
        • Config Management
        • Security Configurations
          • Security - General
          • Security - Source
            • Active Directory
            • Atlassian
            • Azure Active Directory
            • Builtin Security
            • Miro Security
            • Google Directory
            • SharePoint Security
            • Slack Security
          • Security - Sync
          • Security - Agents
          • Security - Scheduling
        • Source Configurations
          • Source - General
          • Source - Source
            • Alfresco Kafka
            • Azure Blob Storage
            • BBC Monitoring
            • Big Query Cataloguer
            • BIM360
            • CSV Data Loader
            • Confluence
            • Content Server
            • Data File Cataloguer
            • Document Store
            • DocuSign
            • Dropbox
            • Exchange 365
            • Filesystem
            • Google Bucket
            • Google Drive
            • Google Vault
            • Jira
            • JSON Data Loader
            • Livelink
            • Microsoft Teams
            • Mimecast
            • Miro
            • ODBC Data Loader
            • PowerBi Cataloguer
            • Reuters Connect
            • ShareFile
            • SharePoint
              • Azure Portal and Azure AD Authentication
              • Sensitivity Labels
            • SharePoint Legacy
            • SQL Server Cataloguer
            • Slack
            • Versioned Document Store
            • Websites
            • XML Data Loader
          • Source - Crawl
          • Source - Agents
          • Source - Schedule
          • Source - Advanced
        • Enrichment Configurations
          • Creating a Pipeline
            • General
            • Steps
              • AccessMiner
              • AI Classification
              • Apply Sensitivity Label
              • Anonymiser
              • CAD Extractor
              • Checksum
              • Content Retrieval
              • Copy
              • Data Rule Processor
              • Delete
              • Direct Copy
              • Email Extractor
              • Entity Rule Processor
              • External Links
              • Geotag
              • Google NLP Extractor
              • Google Vision Extractor
              • Metrics Calculation
              • Microsoft Vision Extractor
              • OcrRest
              • Office Metadata
              • PCI Extractor
              • REST
              • Set Document Risk
              • Text Cleaner
              • Tika Text Extraction
              • Trie Entity Extractor
              • Update Metadata
            • Filters
            • Agents
            • Schedule
            • Advanced
        • OCR Engine
        • Job Configurations
          • General
          • Job
            • AutomatedSearchJob
            • Command Job
            • ElasticJob
            • Extended Metrics Job
            • File Extractor
            • GoogleVaultSAR
            • Google Drive Last Access Date
            • Nightly Events Processor Job
            • Notifications Processor Job
            • Portal Sync Job
            • Purge Job
            • SAR Archiving
            • Text Content Merge Job
          • Output
          • Agents
          • Scheduling
        • Migration Configuration
          • General
          • Filter
          • Metadata Mappings
          • Agents
          • Scheduling
          • Advanced
      • Stats
        • Data Views
    • Security
      • User Settings
      • Credentials
      • Authentication
      • Application Access
      • Auditing
      • Descriptor Groups
      • Uploads
    • Mappings
      • Entities
        • Manage Entity Groups
        • Create an Entity
        • Manage an Entity
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Search Settings
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Bulk Search
        • Managing a Bulk Search
      • Filtering
      • Search Performance
      • Related Results
      • Featured Links
    • AI Settings
      • Search Flows
        • Search Flow Types
        • General Configuration
        • Query and Prompt Classification
        • Search Steps
        • Smart Filtering
        • Model Steps
        • Result Templates
        • System Prompt
      • Tools
        • Concepts
        • Import OOTB Tools
        • Built In Functions and Tools
        • Create and Edit Tools
      • Classifications
        • Class
        • Class Rules
        • AI Classification
    • User Interface
      • Thumbnails
      • Code of Conduct
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Timeline Lens Activity Chart
        • Relationship Map
      • Map Lens
      • Theming
      • User Avatar
    • Global Settings
      • General
      • App Settings
      • Presets
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • SAR Disclosure Document Storage
        • Getting SAR data from Google Vault
        • SAR Configuration Access
        • SAR File Status
      • Disclosure Portal
        • Disclosure Portal Set Up
        • SARs From The Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Collections
      • Notifications
      • OData API
  • AI Studio
    • Classifications
      • Classifications
      • Classification Rules
    • Jobs
  • Labels
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Library
      • Data Science API Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
      • Search API Wrapper
        • Login
        • Privileged Access
        • Search
        • Collection
        • ChatBot Class
      • Admin API Wrapper
      • AI Model Server API Wrapper
      • Utilities
        • Query Builders
        • Azure Key Vault Wrapper
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • Whitepapers and Explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  1. Control Hub
  2. Agents
  3. Configurations
  4. Source Configurations
  5. Source - Source

SharePoint

PreviousShareFileNextAzure Portal and Azure AD Authentication

Last updated 27 days ago

CtrlK
  • Recommended SharePoint API Application Permissions
  • Connection
  • Primary
  • Secondary
  • Sites
  • Permissions
  • Mappings
  • Advanced
  • API Throttling
  • Utility Paths
  • Crawl Options
  • Document Link

Connect your SharePoint source to Aiimi Insight Engine to make the most of your data. Once you have selected a Source System type more detail will expand to customise this.

Recommended SharePoint API Application Permissions

The two recommended permission options for your SharePoint API are Sites.Full.Control.All or Sites.Selected.

Sites.FullControl.All - Allows the app full control of all site collections.

  • This allows add, edit and delete operations on ALL site objects.

  • This can add, edit, delete entire site collections and document libraries.

Sites.Selected - Allows the application to access a subset of sites. The specific sites and the permissions granted are configured for each sites granted identities and must use FullControl.

  • This only works if you grant Sites.Selected for the Registered Application used to connect to SharePoint Online. (Application A).

  • You must then use another Azure Registered Application with Graph API Application permissions of Sites.FullControl.All to add the SharePoint application to each sites granted identities. (Application B)

  • This requires the highest level of permissions to change.

Azure Portal Azure AD Authentication

Azure Communication Services (ACS) are being deprecated. Authentication via Azure Portal and Azure AD is the modern way to manage app registration, communication and authentication.

You can use an Azure Registered Application with a certificate to connect to SharePoint Online. This allows for modern API Permission management scopes such as Sites.Selected via the SharePoint API in Azure.

For support setting this up use our guide on Azure Portal and Azure AD Authentication.


Connection

Primary

  1. Authentication Mode - Select the type of authentication to use when connecting to SharePoint.

  2. Client ID - Enter the SharePoint online or Azure Registered Application Client ID to use.

  3. Directory (Tenant) ID - Enter the SharePoint online or Azure Registered Application Tenant ID to use.

  4. Select Credential (Username & Password) - Choose the credentials to use.

    • For support setting up credentials use our guide on managing credentials.

  5. Select Credential - If your SharePoint Online does not have a Username and Password you must use a secret or certificate. Select the matching SecretOnly or Certificate credential for your SharePoint.

    • This will be the Azure Portal/Azure AD Certificate credential if required.

      • For support setting this up use our guide on Azure Portal and Azure AD Authentication.

Secondary

  1. Use Graph API for Permissions - If checked, the Graph API will be used to retrieve permissions.

  2. Use Graph API for Site Discovery - If checked, the Graph API will be used to retrieve all sites, site collections and one drives.

    • If this is checked the Admin Centre URL won't be used.

  3. Graph API Endpoint - Enter the Graph API endpoint.

    • In most situations the default does not need changing.

  4. Directory (Tenant) ID - Enter the SharePoint online or Azure Registered Application Tenant ID to use.

    • This can be found in Azure Enterprise Application configuration page.

  5. Authentication Endpoint - Enter the endpoint that's needed to authenticate requests.

    • The default does not need changing in most cases.

  6. Select Credential - Select the Client ID secret for SharePoint Online.


Sites

  1. Site Discovery Type - Choose the type of sites should be discovered during crawl.

    • Choose SharePoint, OneDrive, or Both.

  2. Admin Centre URL - Enter the root URL of the site collection that should be crawled.

    • To use the admin centre URL Aiimi Insight Engine requires the highest level of access.

    • Format - https://[site]-admin.sharepoint.com

  3. Sites - Choose the sites within the root collection to crawl.

    • Enter in specific Sites or leave blank to crawl all sites.

    • If you enter a site to crawl the Site Discovery Type setting will be ignored.

  4. Sites to Exclude - If sites need excluding from the Crawl add them to Sites to Exclude.

  5. Included libraries - Add specific libraries to crawl. This list overrides any exclusions if there is an overlap.

  6. Excluded libraries - Libraries can be excluded from the Crawl by adding them to the Excluded Libraries list.

    • The default excluded libraries can be removed or edited as needed.


Permissions

Synchronisation

  1. Security Configuration - Enter the ID of the security configuration the connector will use to synchronise objects with Aiimi Insight Engine Users. This field is required for permission trimming.

    • The ID must match a SharePoint Security Configuration. See our SharePoint security configuration guide for help setting this up.

This field is required if you are using Graph API for permissions.

Groups

  1. Additional Included Groups - Add any additional user groups that should have access to this source within Aiimi Insight Engine.

    • Permissions need to be granted in SharePoint for these users to access any items.

  2. Excluded Groups - Add any user groups that should not have access to this source.


Mappings

  1. Owner Content Type Mapping - Enter the content field type name to be used as the owner of a file.

    • Formatted like SharePoints internal naming conventions. Content_x0020_Owner

  2. Entity to meta tag mappings - You can map the entity fields to meta tags from the web page.

    • Enter the full entity field in the left column. For example, entities.Websites.category.

    • Enter the meta tag name in the right column

  3. Metadata to meta tag mappings - You can map the metadata fields to meta tags from the web page.

    • Enter the full metadata field in the left column. For example, metadata.webtype.

    • Enter the meta tag name in the right column


Advanced

API Throttling

  1. Page Load Throttling - To reduce timeouts you can change the Page Load Throttling.

    • By changing this you can increase the wait time between Chromium Page loads.

    • Enter the time in MS (2000 ms = 2 seconds).

  2. Content Limit Per Request - Enter the maximum number of requests that can be made to Sharepoint.

    • Increasing the limit can improve the crawls performance. But, this is at the risk of creating timeouts if your SharePoint libraries contain complex metadata.

    • Lowering the Content Limit Per Request more requests will run but each run will be quicker.

    • We would recommend starting with a batch size of 500.

  3. SharePoint Client Context Timeout - Set a timeout value for all requests made to Sharepoint.

    • Enter a CSOM value into the SharePoint Client Context Timeout.

    • By default this is set to -1.5.

Utility Paths

  1. Path to SharePoint Online (only) Cookie Utility - Enter the path to your preferred cookie authentication utility.

    • This will handle authentication when processing ASPX pages.

    • You must explicitly enter the full path including the .exe.

    • If this field is left blank Default utils install locations will be used.

  2. Path to Chromium (Chrome) - If Chromium has been manually deployed enter the full path including chrome.exe.

    • If this field is left black it will be downloaded once.

Crawl Options

  1. Process without Delta Tokens - Check this to NOT use the library delta tokens.

    • This will do a reprocess of the full library at every crawl and can be enabled to sync an index. All documents will be retrieved from the library, regardless of changes. However, only necessary changes will be made to your index.

      • If this is not checked, only the changes since the last change token will be retrieved from the library. Likewise only the necessary changes will be made to your index.

    • To use SharePoint change tokens to enable delta crawls uncheck this option.

  2. Build Site Caches - Check to build a cache of all users with access to a site.

    • If unchecked validation will be processed with each document.

  3. Generate Edit in Browser Link - Check to generate an Open In browser app as default (SP Online only).

  4. Get Permissions In Bulk - Check to gather all document permissions for everything in a library at once.

    • The higher the document count the more memory intensive it will be.

    • Not retrieving the permissions in bulk will mean a permission request for every file indexed. This can get you rate limited by SharePoint if many requests are sent in quick succession.

  5. Generate Open in App Link (MS Office docs only) - Check to generate an Open In link as SharePoint does.

    • For example, Open in word on the Users Machine (SP Online).

    • This will override the open in browser option.

Document Link

  1. CSS Class Name - To extract a document link in aspx pages, enter the CSS classname for the hyperlink element.

  2. Document URL - To extract a document link in aspx pages, enter the document link URL.

  3. Metadata Field - To extract a document link in aspx pages, enter the metadata field in the index to store the link.