The Profiling Plugin Settings

The Profiling Plugin Settings

Version 1.1.58.13 (Apr 30, 2025) and later

This article explains how to configure and use the Profiling plugin to assess the quality of your data before transformation or migration.

Overview

The Profiling plugin enables the generation of data profiling reports directly within the app. It helps users assess the source data's quality, completeness, and structure before transforming or transferring it to the target system.

The plugin is built as a reusable extension based on the Plugin Builder Target, preconfigured for profiling scenarios. It leverages the YData Profiling library to generate visual summaries, descriptive statistics, missing value indicators, correlation heatmaps, and alerts on potential anomalies, enabling users to make informed decisions early in the migration process.

Prerequisites

1. You have created a new project.

2. You have added at least one data source plugin to the Design area.

3. You have added the Profiling plugin to your canvas from the Extensions panel.

Configuration

In the Configuration tab, you can customize how your profiling reports are generated.

Required fields:

• Report Title: The title shown in the output file.

• Report Output Directory: Where to save the HTML reports.

• Report Format: Only HTML is currently supported.

Optional sections to include (toggle ON/OFF):

• Interactions

• Correlation types: Pearson, Spearman, Kendall, Cramers V

• Missing value visualizations: Bar Chart, Matrix, Heatmap

• Sample section (e.g., first and last 10 records)

• Duplicate Rows section

You can also set how many rows to include in the Sample section (default is 10).

Figure 1: Configuration

Field settings

Input fields:

• Represent the structure of your incoming data.

• One profiling report is generated for each input table.

• You can add tables and fields manually or let the system auto-generate them using the Field Rendering Script (based on file or database structure).

Output fields:

The plugin generates multiple result tables:

• Report Head Data: Metadata like report name and file path.

• Overview Data: Key statistics such as the number of variables, missing values, duplicate rows, and memory usage.

• Alerts: Any quality issues found in the data, flagged based on custom thresholds.

Figure 2: Fields

Business rules

Use the Business Rules tab to define thresholds that control which alerts appear in your report. These include:

• high_cardinality (default: 50)

• high_correlation_pearson, spearman, kendall, cramers (default: 0.9)

• imbalance (default: 0.5)

• skewness (default: 20)

• chi_squared (default: 0.999)

These thresholds directly influence how YData Profiling detects outliers, correlations, or imbalances.

Figure 3: Business Rules

Field rendering

The Profiling plugin supports automatic field generation based on the structure of your input data. Depending on the source type, specific fields must be configured in the Basic Settings tab:

Mapping

Maintain the required settings for all plugins used in your scheme and map fields as described in Mapping.

Figure 4: Mapping

Running the project

After mapping all necessary fields, save your project. You can then run it manually, schedule it for later, or trigger it based on an event, as described in Running the Project.

Analyzing run results

1. Monitor project execution from the Data Upload screen.

2. Click the link in the Tech Log column to view the generated TXT report per each single row of processed data.

3. Use the Open the Logs Folder button to access CSV log files and track the data upload path during data execution.

4. Check the final report file, which contains:

• The summary of the dataset structure and memory usage 
• Variable-level statistics (e.g., types, unique values)
• Visual charts of correlations and missing data
• Warnings or alerts based on your thresholds

The report is generated in the Project Run Files folder once the project is completed.