Real Time Data Replication Using Qlik Replicate

  • ETL(Extraction Transformation Load) / ELT(Extraction Load Transformation)
  • It is typically used to move data from one location to another with a transformation during the migration.
  • Real time data replication
  • It delivers data on records that changed for database functions such as inserts, updates, and deletes and makes a record of that change available either within a database itself or to other applications that rely on the data in a near real time manner.
  • CDC tools typically rely on the database’s transaction log, which keeps track internally of record changes for system recovery. CDC tools leverage that information to deliver database changes to an external system.
  • The use of timestamps.
  • Database transaction logs.
  • A database triggers log or publish change events to a table and shares those changes with the CDC system.
  • Load data efficiently and quickly to operational data stores/warehouses
  • Create copies of production endpoints
  • Distribute data across endpoints
  • Full Load Replication: Creates files or tables on the target endpoint, automatically defines the metadata that is required on the target, and populates the tables with data from the source
  • Change Processing, also called Change Data Capture (CDC): Captures changes in the source data or metadata as they occur and applies them to the target endpoint as soon as possible, in near-real time.
  • Store Changes: Captures changes in the source data or metadata as they occur but does not apply them to the target.

Qlik Replicate to ingest and deliver data in real-time to the Snowflake environment:

  • Replace the text New Endpoint Connection 1 with something more descriptive like MySQL-Source,
  • make sure the Source radio button is selected,
  • and then choose MySQL from the dropdown selection box.
  • Server: mysqldb
  • Port: 3306
  • User: root
  • Password: **********
  • Security/SSL Mode: None
  • Similarly, create a Snowflake connection.
  • A source endpoint
  • A target endpoint
  • The list of tables that we want to capture
  • Any transformations we want to make on the data.
  • Name: MySQL-to-Snowflake
  • Unidirectional
  • Full Load: enabled (Blue highlight is enabled; click to enable / disable.)
  • Apply Changes: enabled (Blue highlight is enabled; click to enable / disable.)
  • Store Changes: disabled (Blue highlight is enabled; click to enable / disable.)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store