LDAP¶

The sync process will select the users to be sync'd based on a list of LDAP queries that returns groups from the AD tree. Our suggestion is to create a Windows Group called "Sequel Application Users" and include all users to be sync'd in this group. However, it is possible to define custom LDAP queries and also multiple queries.

While being an expert on (https://ldap.com/) is not required; we highly recommend to be familiar with LDAP and understand properly how the AD tree is defined in the catalogue we want to sync from. Some basic concepts are important as: DNs, CN and basic query filters on groups.

Architecture¶

Below diagram describes how the solution integrates with the Sequel's Security Architecture and the AD server.

The three services (not servers) involved in this solution are:

• A Windows Active Directory (AD). Supporting LDAP.
• A "Sequel Security AD Sync" Windows Service (sync service).
• A Sequel's Security API Service (Security API).

The sync service periodically executes a "sync process", this process in requests information of groups and users, based on customized LDAP queries against the AD. This information is translated to an intermediate contract LdapUser that represents a user with some key attributes retrieved from AD, like name or groups that user is member of. The sync service will request the creation/update of this user to the Security API. This request will determine if this is a new user or an existing one, it will calculate the memberships for this user based on the information and it will apply the changes to the user. All actions during the process are audited in the Security API, and error logged to core logging repository (central logging repository used by Sequel Apps).

This solution is an ETL (extract, transform, load) where sync service is responsible of extracting data from AD using LDAP; and Security API is responsible of transforming and loading the data.

On premise deployment¶

The typical deployment of this feature is on a full on-premise installation where Windows AD and Sequel's products are installed on the customer's premises. Below diagram displays this topology, also in this sample authentication is configured using Windows.

Hybrid deployment¶

For cloud deployments, when Windows AD is at customer's premises is still possible to get this synchronization working, the conceptual architecture is described below:

1. A VPN is required to connect the LDAP Sync Service with Windows AD
2. LDAP Sync Service is running hosted in the cloud and performs synchronization, allowing creating and maintenance of users and groups. This is required for Authorization purposes.
3. Authentication in this sample is done with Azure AD. The customer is responsible of synchronizing their Azure AD tenant with their Windows AD. We can assume this as a normal practice for all customers moving their authentication to Azure AD.

Terminology¶

Sync Process¶

Each execution performed to synchronize the Sequel's Security information from an Active Directory.

Each sync process contains below information:

• Basic details: Unique Id, and when the process was started, completed or cancelled.
• Audit information: traces about how the process is behaving during the different phases of the extract, transform and load.
• Users to sync: list of users detected in the AD and the raw information collected and used to match and sync in the Sequel security service.

The sync process contains several models for configuring, executing and tracing the process; that we will be described in this document. As a first summary, all domain model for the sync process is included in the below class diagram:

Above models are domain model, and they are not representing the database models.

MembershipSet¶

Understanding what is and how works MembershipSet is recommended.

Configuration¶

Membership management¶

MembershipSet can be configured from the Security API and from Administration UI at the Membership Set section. More info at MembershipSet management section.

Sync Configuration¶

LdapSyncConfiguration resource in API¶

The sync configuration is managed by Security API service, and is available at LdapSync\Configuration resource (protected by LdapSyncConfiguration securable). This configuration is stored in [authorization].[LdapSyncConfiguration] table in the security database.

If there is not a configuration stored in the system; the GET endpoint returns a default configuration. If PUT is requested and there is not a configuration stored; then a new configuration is stored in the system. If there is already a configuration stored; this configuration is updated.

This resource is used by the Administration UI for configuring the sync process, al settings are described in the next section.

Configuring from UI¶

The configuration can be managed from the Admin UI as well from the System Configuration in the section Role-level Authorization sync with Active Directory.

Clicking on this option we will be able to configure the settings displayed above in below screen:

A LdapSyncConfiguration object contains below properties:

The GroupQuery object contains a single property called Query (string) that contains a LDAP query for retrieving group information. The Vanilla configuration contains a single entry with below LDAP query:

(&(cn=Sequel Application Users))

So, by default all users assigned to the group Sequel Application Users will be sync'd.

The SearchBaseQuery indicates a base domain to be used in the queries. This value must be set if request are done against the global catalog. The value should contain DC entries: if your base domain is office.local, the SearchBaseQuery should be DC=office,DC=local.

The approach followed to configure the LDAP queries reduces highly the risk of a LDAP injection, also this configuration can be done by security admin users so the risk is even lower. However, we recommend to use a service account for accessing Windows AD with read only permissions.

The UserMapping object contains a few properties to define the mappings; Maybe the most important are SsoUsername and Username that must be assigned to SamAccountName or UserPrincipalName; in order to identify the user and define how to map the user. The SsoUsername identifies the user in the AD.

Monitoring¶

At monitoring page, we can review the list of recent sync process executed and its state. By expanding a process, you will be able to check the audit log and the synchronized users.

The monitoring will automatically refresh the information every 10 seconds when the user is in the Monitoring page.

At the audit tab, all steps of the process are logged and some of them provide more details about the process. If any step contains errors, the step indicates this state and provide details about the problem.

The users tab offers information about the users detected in the sync process and the status of synchronization: created, updated, not-changed,...

Clicking on the user icon of each row, the information collected from AD is presented.

Related API endpoints for retrieving information for this screen are:

• GET LdaPSync\Process: returns all process registered by the system.
• GET LdapSync\Process\{syncProcessId}\Audit: returns all audit entries for sync process syncProcessId ordered by AuditAt field.
• GET LdapSync\Process\{syncProcessId}\Users: returns all LdapSyncUser entries for sync process syncProcessId ordered by AuditAt field. This model contains the LdapUser information and some control properties (when was requested the synchronization, the synchronization was completed, has created a new user, has updated an existing user,...).

Force a manual sync¶

Also, from this screen a new manual sync can be forced. Clicking on the "Synchronize" button, a request to API to POST LdapSync\Process\ is sent and the sync process is manually triggered. This functionality requires the service bus; as the process is triggered asynchronously.

Extract: The Sync Service¶

Sync service is a .net core application that can be executed as a console application and also hosted as a Windows Service; and it is responsible of querying the AD server for getting the affected users and push these information to the Security API for reflecting the changes in the Sequel's security schema.

The sync service (sequel-security-sync-ldap.exe) executes the extract process:

• after being started (startup).
• after a scheduled time (polling): scheduled by default to be executed each hour. This value can be customized. The configuration is refreshed from the SecurityAPI during the startup and in each polling/execution.
• a manual request (manually-forced): the process can be manually triggered from the Admin UI (also from Security API). For using this functionality is required to configure the message bus settings to the sync service. A manual request refreshes the configuration and restarts the polling.

Dependencies¶

The sync service has dependencies with:

• Windows AD - LDAP server. (LdapConnection)
• Sequel's Authentication Service
• Sequel's Security API.
• Message Bus. (MessageBusSettings)
• Sequel's Logging repository. (LoggingSettings)

Process description¶

The extract part of the sync process is described with below activity and sequence diagrams.

Activity diagram¶

Sequence diagram¶

Starting the process¶

The process start because the service was started, it was scheduled or a message was received in the bus (Sequel.Security.MessageBus.Contracts.LdapSync.v1.RunProcess). If a request is received during the execution of the process, this request is ignored by the system, sending a Cancel message to the API.

Once started, the extract process is divided in four steps: initialize, detect groups, detect users and push users.

Step 1. Initialize process¶

At this step the process is initialized:

• a unique identifier for the process is generated. The ID will be generated as YYYYMMDDhhmmss-MachineName.
• and, configuration is retrieved from the Security API service. As part of this request, the process is registered in the Security Service.

For starting the process, the sync service will call to POST LdapSync\Process\{SyncProcessId}. This endpoint will create entries on the [authorization].[LdapAudit] table. Once received the response, the sync service requests the current configuration and applies it to perform the process. The polling time can be affected with the refresh of the configuration.

If the sync process is disabled in configuration; the request to create a new one will return bad request; and the sync process will send a request for cancelling the process.

Always, the latest configuration will be requested and applied; even if the process is cancelled. So, the next execution is configured with the latest configuration (waiting time)

Step 2. Detect groups to sync¶

In this phase the groups to be sync'd are detected. The configuration process contains a list of LDAP queries to retrieve the DN of the groups that in the next phase will be used to retrieve the users to be sync'd. A sample query for retrieving groups will look like:

(&(cn=Sequel Application Users))

(&(cn=*Underwriters*))

The configuration contains a list of queries for retrieving DN of groups. All those queries have to be executed. Once all queries have been executed, this has to be notified to the security API calling to POST LdapSync\Process\{SyncProcessId}\Audit where the body contains below contract:

[
    {
        "Level" : "Error" | "Information" | "Warning",
        "Type"  : "DetectGroups" | "DetectUsers" | "PushUsers",
        "Message" : "Brief message describing the error/action",
        "Details" : "Details related to the error/action: query executed, parameters passed,...",
    },
    {
        "Level" : "Error",
        "Type"  : "DetectGroups",
        "Message" : "Error executing query X",
        "Details" : "Error message received when the query was executed"
    },
    {
        "Level" : "Information",
        "Type"  : "DetectGroups",
        "Message" : "Executed query X",
        "Details" : "Found n groups."
    }
]

The DNs retrieved from the queries will be stored by the process in memory managing below requirements:

• Groups members of other groups are not processed. Inheritance of groups is not supported.
• List of groups cannot contain duplicated groups to avoid duplicated request in the next steps.

Step 3. Detect users¶

With the list of DN groups from the previous phase; we will detect the users to be sync'd. The query for retrieving users is defined in the UserQuery property of the configuration; using the default one or the custom one. The query will be validated ensuring the query contains the token {groupDn} . As this token is replaced with the DN groups detected in the previous phase.

The list of attributes requested in this query to AD are:

The user information stored in memory uses the LdapUser model contract used in POST LdapSync\Process\{SyncProcessId}\Users for pushing changes to API. This information is stored in the database for tracing purposes as well at [authorization].[LdapSyncUser] .

Similar to the previous step, this process needs to notify to security API about the current state calling to the same endpoint POST LdapSync\Process\{SyncProcessId}\Audit where the body contains below contract:

[
    {
        "Level" : "Error" | "Information" | "Warning",
        "Type"  : "DetectGroups" | "DetectUsers" | "PushUsers",
        "Message" : "Brief message describing the error/action",
        "Details" : "Details related to the error/action: query executed, parameters passed,...",
    },
    {
        "Level" : "Error",
        "Type"  : "DetectUsers",
        "Message" : "Error executing query for group DN",
        "Details" : "Error message received when the query was executed"
    },
    {
        "Level" : "Information",
        "Type"  : "DetectUsers",
        "Message" : "Detected users for group DN",
        "Details" : "Found n users."
    }
]

The aggregated list of users detected cannot contain duplicated entries, so we can ensure we will not request duplicated syncs for a given user.

Step 4. Push users to Security API¶

At this step, the process is managing a list of users to be sync'd. Following the indications of the configuration, the process creates batches of users and post them to POST LdapSync\Process\{SyncProcessId}\Users, using below contract in the body:

Completing the process¶

When the process is completed, this is notified to the API calling to PATCH LdapSync\Process\{SyncProcessId}\Complete.

In general, only unhandled errors will force to stop the process. In this case, errors are logged to the Sequel.Core.Logging repository; and cancellation will be notified to the Security API (if possible) PATCH LdapSync\Process\{SyncProcessId}\Cancel, including the reason: the error message of the unhandled exception.

Infrastructure settings¶

All infrastructure dependencies are reflected in the appsettings.json file located at the LDAP service installation folder (e.g.: C:\Security\sequel-security-ldap-sync). For further information, see also AD Sync Registration.

{
  "LdapConnection": {
      "Host" : "host-name (string)",
      "Port" : 3268,
      "SecureConnection": true,
      "DN" : "dn username",
      "Password" : "password encrypted"
  },
  "SecurityConnection": {
    "ClientId" : "sec.ldapsync",
    "ClientSecret" : "secret",
  },
  "ServiceDiscoverySettings": {
    "Mode": "PointToPoint",
    "RequiredServices": {
      "SecurityApi": {
        "InternalUrl": "<INTERNAL-URL>",
        "ExternalUrl": "<EXTERNAL-URL>"
      },
      "Authentication": {
        "InternalUrl": "<INTERNAL-URL>",
        "ExternalUrl": "<EXTERNAL-URL>"
      }
    }
  },
  "MessageBusSettings": {
    "Application": "SecurityLdapSync",
    "Instance": "{machine-name}",
    "RabbitMqSettings": {
      "Password": "pwd-encrypted",
      "ServerUri": "rabbitmq://sbsmpormqsmt.office.sbs/SuppAppTests",
      "UserName": "security",
      "PurgeOnStartup": true,
      "Timeout": "0.00:45:00.0000",
      "BindMessageExchanges": true
    },
    "Consumers": [
      {
        "ConsumerType": "Sequel.Security.Tools.LdapSync.Application.Consumers.RunProcessConsumer, Sequel.Security.Tools.LdapSync.Application, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
        "MessageType": "Sequel.Security.MessageBus.Contracts.LdapSync.v1.RunProcess, Sequel.Security.MessageBus.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
        "QueueName": "LdapSyncRunProcess",
        "QueuePerInstance": false
      }
    ]
  },
  "LoggingSettings": {
    "MsSqlLoggingSettings": {
      "LoggingEnabled": true,
      "ConnectionString": "sql-connection-string",
      "IgnoredLogTypes": [],
      "MinimumLogLevel": "Debug"
    },
    "RollingFileLoggingSettings": {
      "LoggingEnabled": true,
      "PathFormat": "C:\\Temp\\log-{Date}.txt",
      "PropertiesTemplate": "{Message}{MessageTemplate}{TimeStamp}{Level}{Exception}{Properties}{LogEvent}{ErrorGuid}{TraceGuid}{SystemName}{ExecutionTime}{MachineName}{LogType}{UserName}",
      "PropertyFormat": "{{PropertyName}:{PropertyValue}}{Tab}",
      "IgnoredLogTypes": [],
      "MinimumLogLevel": "Debug"
    }
  }
}

LdapConnection¶

SecurityConnection¶

Contains information for authenticating and connecting to the Security API, following the standard defined in other components of security services

MessageBusSettings¶

Defined following schema provided by Sequel.Core.MessageBus, and configured with a single consumer Sequel.Security.Tools.LdapSync.Application.Consumers.RunProcessConsumer , for the contract Sequel.Security.MessageBus.Contracts.LdapSync.v1.RunProcess. The API publishes this message with a time to live of 5 minutes by default.

LoggingSettings¶

Defined following schema provided by Sequel.Core.Logging.

Transform: MembershipSet¶

When a request is received to POST LdapSync\Process\{SyncProcessId}\Users, two main actions occurs:

• Transform a LdapUser to a Sequel's users.
• Load User (upsert).

In this section, we will cover the transformation logic.

Transformation logic¶

The transformation logic is function that transforms a LdapUser into a Sequel's User. We have to differentiate two types of information.

Basic information¶

Including information as username, first name, last name or email address. This mapping is defined with default settings, and some of them can be customized. Below table describes the available fields and the default mappings:

The mapping for SsoUsername is quite important; as this will allow the user to SSO using Windows Authentication. But to complete the login, SsoDomain must be defined in the configuration (using UI) so both ${SsoDomain}\${SsoUsername} will form the valid username for the Authentication.

The configuration for mappings (SamAccountName and UserPrincipalName) can be done from the Ldap Configuration API endpoint and associated UI (card "User mappings").

Membership information¶

The memberships translation logic is convered with details at membership set management section.

Load: The Security API Endpoint¶

The transformation and load parts of this ETL sync process are managed in the Security API; including configuration and monitoring functionalities; all defined under the LdapSync resource.

The creation of a new sync process will always audit the request; when the sync is disabled the response will be a BAD REQUEST with a validation error in the body indicating the sync process is disabled.

Process endpoints¶

GET /Process¶

Returns all process audited by the system.

POST /Process¶

Publishes a message in the bus that will be consumed by Sync Service and then the service will start the sync process.

GET /Process/{syncProcessId}¶

Returns information for syncProcessId

POST /Process/{syncProcessId}¶

Registers the start of a new sync process. This will record information in the sync audit tables a new entry of type Started.

DELETE /Process/{syncProcessId}¶

Delete all audit information for process syncProcess (auditing and user information).

PATCH /Process/{syncProcessId}/Complete¶

Completes the process. The underlying action is adding a new entry in Audit with the type Completed. If all process is successfully completed without errors, the detection of deleted users is executed. The audit retention policy is applied here.

PATCH /Process/{syncProcessId}/Cancel¶

Completes the process. The underlying action is adding a new entry in Audit with the type Cancelled. The audit retention policy is applied here.

Sync users¶

Users are stored in the Sequel's AuthZ schema when receiving a request in POST LdapSync\Process\{SyncProcessId}\Users. The logic for this save is described below:

A few rules applies to this process:

• All request are included in the LdapAudit when starting and when completing.
• All LdapUsers included in the request are stored in the LdapSyncUser table; linked to the LdapSyncProcess.
• All LdapUsers have to be processed. If one of them fails, the information is logged and audited; and the process carries on with the next one.
• Users contains a flag to indicate if LDAP Sync is enabled (LdapSyncEnabled bit). If false, users are not synchronized. Also, a new field LdapSyncUpdatedAt: nullable datetime updated each time the user is updated due to the sync process.
• Membership entries contains a flag to indicate when the entry has to be or not included in the synchronization (LdapSyncEnabled). All entries created by the sync process will be flagged as true. Only LdapSyncEnabled entries can be removed during the sync process.
• Users will be updated only when there are changes: basic details or memberships. When the user is created or changed, the related message bus are expected to be published.
• Saving/Creating users/memberships should apply the same rules applied to any action done to users/memberships from API.

Deleted users¶

Once the sync process is completed without errors; the "deleted user detection" will be triggered but only if enabled in configuration. This process will mark as inactive all users with LdapSyncEnabled enabled and were not included in the sync process.

Starting guide¶

For starting working with the role-level based authentication sync with Active Directory we need:

• The LDAP Sync service installed and running (see Installation Guide)
• Role-level based Authentication Sync minimal configuration (see Azure AD AuthN or Azure AD Authentication Registration if necessary). Assuming all default settings are used.
• At least, the domain for the new users created must be populated.
• Depending on Active Directory configurations; it could be required to review some LDAP query settings.
• Membership Sets created and configured in Security; if there are no created Membership Sets created the sync process could create users but they will not have permissions to access.

Appendix¶

Models¶

LdapProcess model¶

Models a Sync Process. Contains below information:

Authentication.User model¶

Changes to user model due to Ldap Sync:

Authentication.Membership model¶

Changes to membership model due to Ldap Sync:

Enabling LDAP Sync on existing installation¶

If we enable LDAP Sync for an existing installation is important to analyse how this sync will affect to existing users. By default, users are flagged to do not sync with LDAP, just those ones created by the automated sync are created with this flag. There are a few points to bear in mind when LDAP Sync is enabled for an existing Security installation:

• Existing users could be flagged with LDAP Sync disabled. Review if those existing users need to be synced. If those users are changed to enable sync, membership must be reviewed as described in the next point.
• Review memberships defined for users with LDAP Sync enabled. In general, users with LDAP sync activated are expected to do not define membership manually (however, there are exceptions where this could be done in purpose).

Performance¶

Some performance tests have been done to determine the expected execution times. The test has been done in a server, with below specs, where all services of security where installed.

The AD server was in the same private network, and the tests was done against a single group with 445 users; where each membershipset was configured with three entries.

LDAP Sync Process was executed with the default settings.

Scenario 1: All new users¶

In this test all users were created..

Scenario 2: Updated 80% users¶

In this scenario, 80% of users were updated due to changes in the memberships.

Scenario 3: Updated 5% users¶

In this scenario 5% users were updated due to changes in the memberships.

Scenario 4: No Changes¶

In this scenario, the most probably, there were no changes detected and users were not changed.

Comparative¶