Data Management Framework

The Data Management Framework (DMF) is the bulk data engine of F&O. When Module 21 talked about asynchronous integration for thousands of records — this is the machinery that makes it happen. Whether you are migrating data from a legacy system, running nightly feeds from a warehouse, or exporting financial data to a reporting system, DMF is where you configure and run it.

The Data Management Workspace

Everything starts in the Data management workspace (SystemAdministration > Workspaces > Data management). This is mission control for all bulk data operations.

Key tiles in the workspace:

Tile	Purpose
Import	Create a new import project (bring data into F&O)
Export	Create a new export project (send data out of F&O)
Job history	View status of all past and running jobs
Data entities	Browse and configure available entities
Framework parameters	Global settings (parallel processing, error thresholds, staging behaviour)
Entity list sequence	Configure import ordering across entities

🔗 Marcus setting up Anita’s migration: “I always start with an export project first — even before the import. Export a few sample records from the target entity so you can see the exact format F&O expects. Then use that file as a template for your import file. Saves hours of debugging field mismatches.”

Import Projects — Step by Step

Creating an import project

1. Data management workspace → Import tile
2. Give the project a name (e.g., “Vendor Master - Phase 1”)
3. Select the source data format — CSV, XML, or Excel
4. Click Add entity → choose the target entity (e.g., Vendors V2)
5. Upload your source file
6. F&O auto-generates the field mapping and shows the staging table
7. Review and adjust mappings → then Import

The three-stage pipeline

Every import flows through three stages:

Source File  →  Staging Table  →  Target Table
  (CSV/XML)      (validation)     (actual F&O data)

• Source → Staging: Raw data is loaded into the entity’s staging table. Format conversion happens here (dates, numbers, enums).
• Staging → Target: DMF runs the entity’s validateWrite(), initValue(), and insert() methods. Business logic executes. Invalid records are flagged.

ℹ️ Why staging tables matter

Staging tables serve as a buffer. If your import fails halfway, the staging data is still there. You can fix the errors and re-run the staging-to-target step without re-uploading the file. This is especially valuable for large imports where re-upload would take significant time.

You can also choose to skip staging for better performance (bypasses the staging step entirely), but you lose the ability to inspect and fix errors. Separately, set-based processing (configured in Framework parameters) uses SQL set operations instead of row-by-row processing during the staging-to-target step — this is a different optimisation that keeps staging but processes records in bulk.

Question

What are the three stages of a DMF import?

Click or press Enter to reveal answer

Answer

1) Source file (CSV/XML/Excel) is loaded into the 2) Staging table (where format conversion and initial validation happen), then pushed to the 3) Target table (actual F&O tables, where entity business logic executes — validateWrite, initValue, insert).

Click to flip back

Field Mapping

Field mapping controls how source columns map to staging/target fields. DMF auto-maps by matching column names, but you often need to adjust.

Mapping types

Mapping Type	Description
Auto-generated	DMF matches source column names to entity field names
Manual	You drag/drop or configure individual field mappings
Default value	Set a constant value for a field (useful for dataAreaId or status fields)
Transform	Apply a value transformation function during mapping

🔗 Marcus’s mapping tip: “The most common mapping mistake? Forgetting dataAreaId. If your source file doesn’t have a legal entity column, set it as a default value in the mapping — otherwise every record imports into the wrong company or fails.”

Mapping visualiser

The mapping screen shows a visual canvas: source fields on the left, staging/target fields on the right, with lines connecting mapped pairs. Unmapped source fields are ignored. Required target fields without a mapping cause validation errors.

Key mapping operations:

• Generate mapping — auto-map based on column names
• Map to entity — skip staging, map directly source → target
• Modify mapping — adjust individual connections
• Default value — set constants for fields not in the source

💡 Exam tip: Mapping source → staging → target

The exam may present a scenario where a field name in the CSV does not match the entity field name. The answer is to use the field mapping visualiser to manually connect the source column to the correct staging field. Auto-mapping only works when names match exactly. Also remember: you can map to staging OR directly to entity — mapping to staging is the default and recommended approach.

Entity Sequencing

When importing multiple entities, order matters. You cannot import sales order lines before sales order headers. Customer addresses need the customer to exist first.

Configuring sequence

In the Entity list sequence area of the Data Management workspace:

1. Add all entities involved in the import
2. Set the execution order (lower number = imported first)
3. DMF processes entities sequentially in the specified order

Common sequences:

Order	Entity	Why
1	Legal entities	Everything depends on company codes
2	Customers	Must exist before addresses or transactions
3	Customer addresses	References the customer record
4	Customer bank accounts	References the customer record
5	Open balances	References customers and accounts

🔗 Marcus’s sequencing lesson: “Tomoko’s first migration attempt failed because she imported vendor invoices before the vendors existed. 12,000 records — all failed with ‘Vendor account not found’. Set the sequence once, save yourself the headache.”

Question

Why is entity sequencing important in DMF imports?

Click or press Enter to reveal answer

Answer

Entity sequencing ensures parent records are imported before child records. Without it, dependent records fail because their foreign key references don't exist yet (e.g., importing addresses before the customer record they reference).

Click to flip back

Composite Entities

Some business documents have a header-line structure — a sales order has one header and many lines. Composite entities bundle these into a single importable unit.

How they work

A composite entity wraps:

• One header entity (e.g., SalesOrderHeaderV2)
• One or more line entities (e.g., SalesOrderLine)

When you import via a composite entity:

• The data file contains both header and line data (nested XML, or separate CSV sheets)
• DMF imports the header first, then the lines — maintaining referential integrity
• It is atomic — if lines fail, the header is rolled back too

Format note: Composite entities work best with XML format because XML naturally represents nested structures. CSV requires separate files for header and lines.

Regular Entity vs Composite Entity

Aspect	Regular Entity	Composite Entity
Structure	Flat — one table/view	Nested — header + lines
Import format	CSV, XML, or Excel	XML preferred (nested structure)
Atomicity	Per-record	Per-document (header + all lines)
Use case	Master data (customers, vendors)	Documents (orders, invoices, journals)
Sequencing	Manual (entity list sequence)	Automatic (header before lines)
Example	CustomersV3	SalesOrderHeaderV2Entity (composite)

Question

What is a composite entity in DMF?

Click or press Enter to reveal answer

Answer

A composite entity bundles a header entity and one or more line entities into a single importable unit. It handles the parent-child sequencing automatically and provides document-level atomicity — if lines fail, the header rolls back too. Best used with XML format for nested structures.

Click to flip back

Change Tracking for Incremental Exports

Instead of exporting all 500,000 customer records every night, change tracking lets you export only records that changed since the last export.

How it works

1. Enable change tracking on the data entity (Data entities tile → select entity → Change tracking tab)
2. F&O uses SQL Server change tracking on the entity’s backing tables
3. When you run an export, DMF queries only rows where SYS_CHANGE_VERSION > last_sync_version
4. Result: a delta export containing only inserts, updates, and deletes since last run

Change tracking modes:

Mode	Description
Entire entity	Track changes on the primary table only
Primary table and related tables	Track the entity’s primary table plus all joined tables — detects changes in any source table
Custom query	Custom query filter applied to determine which records are tracked — narrows scope but uses the same SQL change tracking infrastructure

🔗 Marcus to Tomoko: “Your nightly product export was taking 45 minutes because it exported everything every time. I enabled change tracking — now it only picks up the 200-300 products that actually changed. Export drops to under 2 minutes.”

💡 Exam tip: Change tracking scope

The exam may ask about the difference between “Entire entity” and “Primary table and related tables” change tracking. If a customer’s address changes (stored in a different table joined by the entity), “Entire entity” will NOT detect it because it only watches the primary table. You need “Primary table and related tables” to catch changes in any table that feeds the entity.

Recurring Data Jobs

Recurring data jobs automate import/export on a schedule — no human intervention needed.

Setting up a recurring export

1. Create an export project in the Data Management workspace
2. Select your entity and configure mappings
3. Click Create recurring data job
4. Set the recurrence pattern (every N minutes/hours/daily)
5. Configure the application queue — this is where external systems dequeue the data via API

Setting up a recurring import (enqueue)

1. Create an import project
2. Configure the entity and mappings
3. Create a recurring data job
4. External systems call the enqueue API to submit data files
5. DMF processes them on the configured schedule

The API endpoints:

// Enqueue (import)
POST /api/connector/enqueue/activity-id

// Dequeue (export)
GET /api/connector/dequeue/activity-id

// Acknowledge (confirm receipt of dequeued data)
POST /api/connector/ack/activity-id

🔗 Marcus’s integration pattern: “The dequeue/acknowledge handshake is critical. Your middleware calls dequeue to get the file, processes it, then calls acknowledge to tell F&O ‘I got it, you can remove it from the queue’. Skip the acknowledge and F&O keeps serving the same file forever.”

Question

What are the three recurring integration API endpoints?

Click or press Enter to reveal answer

Answer

1) Enqueue — POST to submit data files for import. 2) Dequeue — GET to retrieve exported data files. 3) Acknowledge (Ack) — POST to confirm receipt of dequeued data so F&O removes it from the queue. Skipping Ack causes the same file to be served repeatedly.

Click to flip back

Aggregate Entities

Aggregate entities are special data entities built on aggregate measurements (pre-computed OLAP-style data). They expose summarised/aggregated data for reporting integrations.

• Built on aggregate data sources (not regular tables)
• Read-only — used for export/reporting only
• Refresh their data based on measurement refresh schedules
• Ideal for integrating with data warehouses or BI tools that need pre-aggregated numbers

ℹ️ Aggregate vs regular entities

Regular entities expose transactional data (individual records). Aggregate entities expose pre-computed metrics — think “total sales by region by month” rather than individual sales orders. The exam may test whether you know aggregate entities are read-only and cannot be used for import.

Exam Practice

Knowledge Check

Tomoko needs to import sales orders with their lines from an external system. Each order has one header and multiple lines. What should Marcus recommend?

AImport the header entity and line entity separately with proper sequencingBUse a composite entity that wraps the header and line entitiesCUse OData $batch to create headers and lines in one callDImport lines first, then patch them with the header reference

Check Answer

Knowledge Check

An export job is running nightly but takes 90 minutes to export 800,000 product records, most of which haven't changed. What should be enabled to reduce export time?

ASet-based processing in Framework parametersBChange tracking on the product data entityCParallel processing with more threadsDComposite entity for the product

Check Answer

Knowledge Check

After configuring a recurring export, the external system calls dequeue and receives the data file. What must happen next to prevent F&O from re-serving the same file?

ADelete the recurring data jobBCall the acknowledge (ack) endpointCMark the export job as completed in the workspaceDNothing — F&O automatically removes it after dequeue

Check Answer

Knowledge Check

A customer address is stored in a separate table joined by the Customers data entity. Change tracking is set to 'Entire entity'. An address is updated. Will the customer appear in the next incremental export?

AYes — change tracking detects changes in all joined tablesBNo — 'Entire entity' only tracks the primary tableCYes — but only if the address table has its own change tracking enabledDNo — address changes are never tracked by data entities

Check Answer

---

Next up: Custom Services, REST & SOAP — build your own service endpoints, consume external REST APIs from X++, and use Electronic Reporting as an integration tool.