Sunday, September 27, 2020

Implementing Dynamic Data Masking in SQL Server

Implementing Dynamic Data Masking in SQL Server

Introduction to Dynamic Data Masking (DDM)

Dynamic Data Masking (DDM) is a technique that restricts access to sensitive data by masking it for users without the required privileges. This approach makes it easier to design and implement security measures in your applications, ensuring that sensitive information is protected while allowing authorized users to access necessary data.

Permissions Required for Dynamic Data Masking

To create a table with dynamic data masking, you don’t need special permissions beyond the standard CREATE TABLE and ALTER permissions for the schema. However, there are additional permissions needed for manipulating the data masks.

  1. Modifying Masks: You need the ALTER ANY MASK permission, in addition to ALTER on the table.
  2. Viewing Data: Users with SELECT permissions can see only the masked data. Assign UNMASK permission to allow viewing unmasked data.
  3. Control Permissions: The CONTROL permission includes both ALTER ANY MASK and UNMASK.

Limitations and Restrictions of Data Masking

There are some column types that cannot be masked, including:

  • Encrypted columns (Always Encrypted)
  • FILESTREAM columns
  • Sparse columns that are part of a column set
  • Computed columns: If a computed column depends on a masked column, it will show masked data.
  • Columns involved in a FULLTEXT index

Example: Creating a Table with a Mask

You can create a table with dynamic data masking as shown below:

CREATE TABLE MembershipMask  
(
  MemberID int IDENTITY PRIMARY KEY,  
  FirstName varchar(100) MASKED WITH (FUNCTION = 'partial(1,"XXXXXXX",0)') NULL,  
  LastName varchar(100) NOT NULL,  
  Phone varchar(12) MASKED WITH (FUNCTION = 'default()') NULL,  
  Email varchar(100) MASKED WITH (FUNCTION = 'email()') NULL  
);

Insert sample data:

INSERT MembershipMask (FirstName, LastName, Phone, Email) VALUES  
('Vipul', 'Bhatt', '456.123.4567', 'Vipul@Himanshupatel.in'),  
('Dev', 'Shah', '456.123.3333', 'Dev@Himanshupatel.in'),  
('Raj', 'Patel', '555.123.4569', 'raj@Himanshupatel.in');

Querying the table will display masked data:

SELECT * FROM MembershipMask;

Example output for a user with SELECT permission:

MemberID FirstName LastName Phone Email
1        VXXXXXXX  Bhatt     xxxx  VXXX@XXXX.com
2        DXXXXXXX  Shah      xxxx  DXXX@XXXX.com
3        RXXXXXXX  Patel     xxxx  RXXX@XXXX.com

Viewing the Data Masking Definition

Use the following query:

SELECT TBLS.name AS TableName, MC.NAME AS ColumnName, MC.is_masked AS IsMasked, MC.masking_function AS MaskFunction  
FROM sys.masked_columns AS MC  
JOIN sys.tables AS TBLS ON MC.object_id = TBLS.object_id  
WHERE is_masked = 1;

Dropping a Mask

To remove a mask from a column:

ALTER TABLE Membership  
ALTER COLUMN LastName DROP MASKED;

Adding or Editing a Mask on an Existing Column

ALTER TABLE Membership  
ALTER COLUMN LastName ADD MASKED WITH (FUNCTION = 'partial(2,"XXX",0)');

Granting and Revoking Mask Permissions

GRANT UNMASK TO TestUserins;
GO
REVOKE UNMASK FROM TestUserins;
GO

Implementing Partial Masking

To reveal only the last 4 digits of a national ID number:

ALTER TABLE EmpInfo  
ALTER COLUMN NatID NVARCHAR(15)  
MASKED WITH (FUNCTION = 'partial(0, "xxxxx", 4)') NOT NULL;  
-- e.g., 716374314 becomes xxxxx4314

Implementing Random Masking

To mask a monetary value within a range:

ALTER TABLE EmpInfo  
ALTER COLUMN SalesYTD MONEY  
MASKED WITH (FUNCTION = 'random(101, 999)') NOT NULL;

Further Reading

For more information, refer to the official documentation:

Microsoft Docs on Dynamic Data Masking

Saturday, September 26, 2020

Database Email and system objects

Database Email and system objects


Database Mail allows your database applications to send emails to users. These emails can include query results and files from your network.

Database Mail operates with two configuration objects: these allow you to set up the necessary settings for sending emails from your database applications or the SQL Server Agent.

Database Mail Design

Database Mail Accounts

A Database Mail account stores the details used by Microsoft SQL Server to send emails through an SMTP server. Each account is linked to a specific email server.

Create a New Database Mail Account with SMTP Details

sysmail_add_account_sp  [ @account_name = ] 'account_name',  
    [ @email_address = ] 'email_address' ,  
    [ [ @display_name = ] 'display_name' , ]  
    [ [ @replyto_address = ] 'replyto_address' , ]  
    [ [ @description = ] 'description' , ]  
    [ @mailserver_name = ] 'server_name'  
    [ , [ @mailserver_type = ] 'server_type' ]  
    [ , [ @port = ] port_number ]  
    [ , [ @username = ] 'username' ]  
    [ , [ @password = ] 'password' ]  
    [ , [ @use_default_credentials = ] use_default_credentials ]  
    [ , [ @enable_ssl = ] enable_ssl ]  
    [ , [ @account_id = ] account_id OUTPUT ]

Database Mail Profiles

A Database Mail profile is a group of related Database Mail accounts. Applications that send mail via Database Mail should specify a profile rather than individual accounts.

Profiles also allow database administrators to manage email access. Users need to be a member of the DatabaseMailUserRole to send email via Database Mail.

Profiles can either be public or private.

Create a New Database Mail Profile

sysmail_add_profile_sp [ @profile_name = ] 'profile_name'  
    [ , [ @description = ] 'description' ]  
    [ , [ @profile_id = ] new_profile_id OUTPUT ]

Add an Account to a Database Mail Profile

After creating a Database Account with sysmail_add_account_sp and a Database Profile with sysmail_add_profile_sp, you can use sysmail_add_profileaccount_sp to associate them.

Database Mail Profile Account
EXECUTE msdb.dbo.sysmail_add_profileaccount_sp  
    @profile_name = 'AdventureWorks Administrator',  
    @account_name = 'Audit Account',  
    @sequence_number = 1 ;

Retrieve Database Mail Configuration

sysmail_help_configure_sp
sysmail_help_principalprofile_sp 
sysmail_help_profile_sp
sysmail_help_profileaccount_sp

Set Database Mail Configuration

The sysmail_configure_sp stored procedure modifies configuration settings for Database Mail at the SQL Server instance level.

Parameter NameValueDescription
AccountRetryAttempts1Number of retry attempts for a mail server
AccountRetryDelay60Delay between retry attempts
DatabaseMailExeMinimumLifeTime600Minimum process lifetime (seconds)
DefaultAttachmentEncodingMIMEDefault attachment encoding type
LoggingLevel2Logging level for Database Mail (normal - 1, extended - 2, verbose - 3)
MaxFileSize1000000Maximum file size for attachments
ProhibitedExtensionsexe,dll,vbs,jsExtensions not allowed in email attachments
sysmail_configure_sp [ [ @parameter_name = ] 'parameter_name' ]  
    [ , [ @parameter_value = ] 'parameter_value' ]  
    [ , [ @description = ] 'description' ]

Send an Email Using Database Mail

This procedure sends an email with options for attaching files or query results.

sp_send_dbmail [ [ @profile_name = ] 'profile_name' ]  
    [ , [ @recipients = ] 'recipient_email' ]  
    [ , [ @copy_recipients = ] 'cc_email' ]  
    [ , [ @blind_copy_recipients = ] 'bcc_email' ]  
    [ , [ @from_address = ] 'sender_email' ]  
    [ , [ @reply_to = ] 'reply_to_email' ]  
    [ , [ @subject = ] 'email_subject' ]  
    [ , [ @body = ] 'email_body' ]  
    [ , [ @body_format = ] 'html' ]  
    [ , [ @importance = ] 'high' ]  
    [ , [ @sensitivity = ] 'normal' ]  
    [ , [ @file_attachments = ] 'file_path' ]  
    [ , [ @query = ] 'sql_query' ]  
    [ , [ @execute_query_database = ] 'database_name' ]  
    [ , [ @attach_query_result_as_file = ] 1 ]  
    [ , [ @query_attachment_filename = ] 'file_name' ]  
    [ , [ @query_result_separator = ] ',' ]  
    [ , [ @exclude_query_output = ] 0 ]  
    [ , [ @append_query_error = ] 1 ]  
    [ , [ @query_no_truncate = ] 1 ]  
    [ , [ @query_result_no_padding = ] 1 ]  
    [ , [ @mailitem_id = ] mailitem_id ] [ OUTPUT ]

Enable Database Mail

sp_configure 'show advanced', 1; 
GO
RECONFIGURE;
GO

sp_configure 'Database Mail XPs', 1;
GO
RECONFIGURE
GO

Troubleshooting queries:

select * from msdb.dbo.sysmail_sentitems
select * from msdb.dbo.sysmail_unsentitems
select * from msdb.dbo.sysmail_faileditems
SELECT * FROM msdb.dbo.sysmail_event_log;

SELECT [sysmail_server].[account_id],
       [sysmail_account].[name] AS [Account Name],
       [servertype],
       [servername] AS [SMTP Server Address],
       [Port]
FROM [msdb].[dbo].[sysmail_server]
     INNER JOIN [msdb].[dbo].[sysmail_account] ON [sysmail_server].[account_id] = [sysmail_account].[account_id];

SELECT is_broker_enabled FROM sys.databases WHERE name = 'msdb';
GO
SELECT * FROM sys.configurations WHERE name = 'Database Mail XPs'
GO

SELECT * 
FROM msdb.dbo.sysmail_profileaccount pa 
INNER JOIN msdb.dbo.sysmail_profile p ON pa.profile_id = p.profile_id 
INNER JOIN msdb.dbo.sysmail_account a ON pa.account_id = a.account_id

SMTP server details for co

Sunday, September 20, 2020

How to find the missing foreign key indexes

Finding Missing Indexes on Foreign Key Relation Tables

In this article, I'm exploring how to identify missing indexes in tables with foreign key relationships.

SELECT DB_NAME() AS DBName
    , rc.Constraint_Name AS FK_Constraint
    , ccu.Table_Name AS FK_Table
    , ccu.Column_Name AS FK_Column
    , ccu2.Table_Name AS ParentTable
    , ccu2.Column_Name AS ParentColumn
    , I.Name AS IndexName
    , CASE WHEN I.Name IS NULL
        THEN 'IF NOT EXISTS (SELECT * FROM sys.indexes
                    WHERE object_id = OBJECT_ID(N''' + RC.Constraint_Schema + '.' + ccu.Table_Name + ''') 
            AND name = N''IX_' + ccu.Table_Name + '_' + ccu.Column_Name + ''') ' + '
            CREATE NONCLUSTERED INDEX IX_' + ccu.Table_Name + '_' + ccu.Column_Name + ' 
            ON ' + rc.Constraint_Schema + '.' + ccu.Table_Name + '( ' + ccu.Column_Name + ' ASC ) 
            WITH (PAD_INDEX = OFF
            ,STATISTICS_NORECOMPUTE = OFF
            ,SORT_IN_TEMPDB = ON
            ,IGNORE_DUP_KEY = OFF
            ,DROP_EXISTING = OFF, ONLINE = ON);'
        ELSE ''
    END AS SQL
FROM information_schema.referential_constraints RC
JOIN INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE ccu ON rc.CONSTRAINT_NAME = ccu.CONSTRAINT_NAME
JOIN INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE ccu2 ON rc.UNIQUE_CONSTRAINT_NAME = ccu2.CONSTRAINT_NAME
LEFT JOIN sys.columns c ON ccu.Column_Name = C.name
    AND ccu.Table_Name = OBJECT_NAME(C.OBJECT_ID)
LEFT JOIN sys.index_columns ic ON C.OBJECT_ID = IC.OBJECT_ID
    AND c.column_id = ic.column_id
    AND index_column_id = 1
-- index found has the foreign key
-- as the first column 
LEFT JOIN sys.indexes i ON IC.OBJECT_ID = i.OBJECT_ID
    AND ic.index_Id = i.index_Id
WHERE I.name IS NULL
ORDER BY FK_table
    ,ParentTable
    ,ParentColumn;

Other References:

SQL Server Functions

Retrieve ASCII Code Value

ASCII ('A') = 65 
ASCII ('BEE') = 66 

Convert Integer ASCII Code to Character

CHAR (65) = 'A'
CHAR (1000) = NULL

Search for a Character

CHARINDEX ('A', 'B', [, S])
CHARINDEX ('Z', 'HELLO') = 0
CHARINDEX ('H', 'HELLO') = 1
CHARINDEX ('OR', 'WORLD') = 2
CHARINDEX ('L', 'HELLO', 4) = 4

Replace a Character

REPLACE ('A', 'B', 'C')
REPLACE('HELLOW',NULL,'')=NULL
REPLACE('HELLOW','','_')='HELLOW'
REPLACE('HELLOW','ELLOW','I')='HI'
REPLACE('HELLOW','L',1) = 'HE11OW'

PATINDEX ('%pattern%' , ‘A’)

PATINDEX('H','HELLO') = 0
PATINDEX('H%','HELLO') = 1
PATINDEX('%L_%','HELLO') = 3
PATINDEX('%L_','HELLO') = 4
PATINDEX ('Z','HELLO') = 0
PATINDEX('%A[0-9]%','AA1A') = 2
PATINDEX('%L[^L]%','HELLO') = 4

Using STUFF to Replace Substring

STUFF('HELLOW',2,5,'I') = 'HI'
STUFF('HELLOW',2,15,'I') = 'HI '  
STUFF('HELLOW',20,1,'I') = NULL
STUFF('HELLOW',0,1,'I') = NULL
STUFF('HELLOW',1,0,'I') = 'IHELLOW'
STUFF('HELLOW',2,5,'I123456')  = 'HI123456'
STUFF('HELLOW ',2,2,'I') = 'HILOW'

Soundex Function

-- Returns a four-character SOUNDEX code for string similarity
SOUNDEX ('Smith') = 'S530'
SOUNDEX ('Smythe') = 'S530'

DIFFERENCE ('A', 'B') of Soundex

-- Calculates the difference between two soundex codes
DIFFERENCE('GREEN','GREENE') = 4

LEN Function - Length of a String

LEN ('HELLOW WORLD') = 11
LEN ('HELLOW ') = 6
LEN (12) = 2
LEN ('') = 0

Extract a Substring

-- Returns L characters starting at position S in string A
SUBSTRING('HELLOW',2,1) = 'E'
SUBSTRING('HELLOW',5,10) = 'OW'
SUBSTRING('HELLOW',10,1) = ''
SUBSTRING('HELLOW',0,1) = ''
SUBSTRING('HELLOW',1,0) = ''

Extract Right/Left Characters

RIGHT ('', 1) = ''
LEFT ('HI', 0) = ''
RIGHT ('HI', 3) = 'HI'
RIGHT ('HELLOW WORLD',5) = 'WORLD'
LEFT ('HELLOW WORLD', 6) = 'HELLOW'

Insert Spaces

SPACE(5) = '     '

REPLICATE Function

REPLICATE ('*', 4) = '****'
REPLICATE ('-', 0) = ''
REPLICATE ('^', NULL) = NULL

Convert Number to String

STR (2.234) = ' 2'
STR (2.234, 4) = '   2'
STR (2.234, 4, 2) = '2.23'
STR (2.234, 6, 2) = '  2.23'
STR (123456.789,4,2) = '****'
STR(123456.789,14,2) = '      123456.79'

CONCAT Function

CONCAT('HELLO')=ERROR
CONCAT(NULL, NULL)= ''
CONCAT('Hi',' ','Patel')='Hi Patel'
CONCAT(12,NULL,34) = '1234'
CONCAT(2014,12,31) = '20141231'

Case Change Functions

LOWER ('SQL') = 'sql'
UPPER ('hi') = 'HI'

Trim Whitespace from Strings

LTRIM (' HI ') = 'HI '
LTRIM (' ') = ''
RTRIM(' HI ') = ' HI'

QUOTENAME Function

QUOTENAME('H NAME') = [TB NAME]
QUOTENAME('H NAME', '] ') =[H NAME]
QUOTENAME('H NAME', '\"') = "H NAME"
QUOTENAME('abc[]dff') = [abc[]]dff]
QUOTENAME('TB NAME', '''')='TB NAME'

REVERSE Function

REVERSE('HELLOW') = 'WOLLEH'
REVERSE(12) = 21

Saturday, September 19, 2020

Track data changes using CDC in SQL Server Enterprise edition

SQL Server offers two features that monitor changes in a database: Change Data Capture and Change Tracking. These features help applications identify Data Manipulation Language (DML) changes such as insertions, updates, and deletions made to user tables in a database.

Change Data Capture Configuration Settings

? maxtrans* -- Specifies the maximum number of transactions to process per scan cycle.
? maxscans* -- The maximum number of scan cycles to be executed to extract all rows from the log.
? continuous* -- A flag indicating whether the capture job should run continuously (1) or in one-time mode (0). For further details, refer to sys.sp_cdc_add_job (Transact-SQL).*
? pollinginterval* -- Defines the interval in seconds between log scan cycles.
? retention** -- Specifies how long change rows will be stored in the change tables.
? threshold -- The maximum number of deletions allowed in a single cleanup statement.

Enabling Change Data Capture for a Database

USE MyDB  
GO  
EXEC sys.sp_cdc_enable_db  
GO

Disabling Change Data Capture for a Database

USE MyDB  
GO  
EXEC sys.sp_cdc_disable_db  
GO

Enabling Change Data Capture for a Table

  
EXEC sys.sp_cdc_enable_table  
@source_schema = N'dbo',  
@source_name = N'MyTable',  
@role_name = N'MyRole',  
@filegroup_name = N'MyDB_CT',  
@supports_net_changes = 1  
GO

Viewing CDC Job Configurations

EXEC sys.sp_cdc_help_jobs

Modifying CDC Job Settings

EXECUTE sys.sp_cdc_change_job  
@job_type = 'cleanup',  
@retention = 86400; -- 60 days

Getting Capture_Instance Name

EXEC sys.sp_cdc_help_change_data_capture

Example 1: Reading Changed Columns

USE AdventureWorks2014

DECLARE @from_lsn binary (10), @to_lsn binary (10)

SET @from_lsn = sys.fn_cdc_get_min_lsn('HumanResources_Shift')
SET @to_lsn = sys.fn_cdc_get_max_lsn()

SELECT * FROM cdc.fn_cdc_get_all_changes_HumanResources_Shift(@from_lsn, @to_lsn, 'all')
ORDER BY __$seqval

Example 2: Reading Changed Columns

USE AdventureWorks2014

DECLARE @from_lsn binary (10), @to_lsn binary (10)

SET @from_lsn = sys.fn_cdc_get_min_lsn('HumanResources_Shift')
SET @to_lsn = sys.fn_cdc_get_max_lsn()

SELECT * FROM cdc.fn_cdc_get_all_changes_HumanResources_Shift(@from_lsn, @to_lsn, 'all')
ORDER BY __$seqval

Check for changes:

SELECT *  
FROM HumanResources.Shift

SELECT *  
FROM cdc.HumanResources_Shift_CT

Checking If CDC Is Already Enabled for a Database

SELECT [name], database_id, is_cdc_enabled  
FROM sys.databases

Checking If CDC Is Already Enabled for a Table

SELECT [name], is_tracked_by_cdc  
FROM sys.tables

CDC System Tables

? cdc.captured_columns - Lists the columns captured.
? cdc.change_tables - Displays all tables enabled for change capture.
? cdc.ddl_history - Records all DDL changes since CDC was enabled.
? cdc.index_columns - Shows the indexes associated with change tables.
? cdc.lsn_time_mapping - Maps LSN numbers.

https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/track-data-changes-sql-server?view=sql-server-ver15

https://docs.microsoft.com/en-us/sql/relational-databases/system-functions/cdc-fn-cdc-get-all-changes-capture-instance-transact-sql?view=sql-server-ver15

How to work with Temporal Tables in SQL Server

How to work with Temporal Tables in SQL Server

SQL Server 2016 introduced the support for temporal tables (also referred to as system-versioned temporal tables). This feature enables you to track historical data by capturing not only the current data but also the state of data at any given point in time. Temporal tables are a significant addition to databases, based on the ANSI SQL 2011 standard.

SELECT name, object_id, temporal_type, temporal_type_desc FROM sys.tables;
temporal_type temporal_type_desc
0 NON_TEMPORAL_TABLE
1 HISTORY_TABLE
2 SYSTEM_VERSIONED_TEMPORAL_TABLE
SELECT OBJECT_NAME(object_id) AS table_name, * 
FROM sys.periods
table_name name period_type period_type_desc object_id start_column_id end_column_id
Employee SYSTEM_TIME 1 SYSTEM_TIME_PERIOD 581577110 7 8
SELECT name AS 'Column_Name', generated_always_type, generated_always_type_desc 
FROM sys.columns

List all temporal tables along with their history tables.

SELECT schema_name(t.schema_id) AS temporal_table_schema, 
       t.name AS temporal_table_name,
       schema_name(h.schema_id) AS history_table_schema, 
       h.name AS history_table_name, 
       CASE 
         WHEN t.history_retention_period = -1 
         THEN 'INFINITE' 
         ELSE CAST(t.history_retention_period AS VARCHAR) + ' ' + t.history_retention_period_unit_desc + 'S' 
       END AS retention_period 
FROM sys.tables t 
LEFT OUTER JOIN sys.tables h ON t.history_table_id = h.object_id 
WHERE t.temporal_type = 2 
ORDER BY temporal_table_schema, temporal_table_name

Creating a table with a history table.

CREATE TABLE dbo.Employee 
( 
  [EmployeeID] int NOT NULL PRIMARY KEY CLUSTERED, 
  [Name] nvarchar(100) NOT NULL, 
  [Position] varchar(100) NOT NULL, 
  [Department] varchar(100) NOT NULL, 
  [Address] nvarchar(1024) NOT NULL, 
  [AnnualSalary] decimal(10,2) NOT NULL, 
  [ValidFrom] datetime2 GENERATED ALWAYS AS ROW START, 
  [ValidTo] datetime2 GENERATED ALWAYS AS ROW END, 
  PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo) 
) 
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.EmployeeHistory));

The script above will create two tables: Employee and EmployeeHistory.

Inserting and updating data in the tables above.

INSERT INTO Employee (EmployeeID, Name, Position, Department, Address, AnnualSalary) 
SELECT 1, 'himanshu', 'dba', 'IT', 'BC', 4000;
GO
INSERT INTO Employee (EmployeeID, Name, Position, Department, Address, AnnualSalary) 
SELECT 100, 'himanshu', 'dba', 'IT', 'BC', 4000;
GO
UPDATE Employee SET AnnualSalary = 5000 WHERE EmployeeID = 1;
GO
UPDATE Employee SET AnnualSalary = 4500 WHERE EmployeeID = 100;
GO
SELECT * FROM Employee;
SELECT * FROM EmployeeHistory;
GO
SELECT * FROM Employee 
FOR SYSTEM_TIME 
BETWEEN '2014-01-01 00:00:00.0000000' AND '2020-11-01 00:00:00.0000000';

Here is the result of the queries above:

Popular Posts