3 Reasons Teradata Migration Will Fail

by: Mike Waas

3 Reasons Why Your Teradata Migration Will Likely Fail – And What You Can Do About It

“My database migration project is way over budget and running late by years!” is not only frequently uttered but also triple-redundant according to Gartner’s Adam Ronthal. The analyst firm suggests that over 60% of all database migrations fail. That is, a migration is started, then starts slipping, overruns the original budget by multiples, until finally management pulls the plug. Time is lost, money wasted, and careers wrecked.

Think 60% is already damning? Turns out, if the legacy system happens to be Teradata the failure rate is significantly higher. While there is no hard data available, anecdotal evidence puts the likelihood of failure of a Teradata migration north of 95%. Successful projects in this category may be the stuff of legends, but they remain extremely rare exceptions in practice.

This is in stark contrast to the fact that practically every enterprise that uses Teradata today is exploring options to replatform away from it. Most of these replatform initiatives are triggered by an enterprise-wide mandate to move to the cloud and tear down data silos. Enterprises are looking to the cloud as a highly innovative environment for their data processing needs. A data warehouse appliance, physical or virtual, no longer fits into this picture.

The desire to leave Teradata, paired with the magnetism of cloud data warehouses like Azure Synapse Analytics or Amazon Redshift, creates an almost perfect force field. Migrating away from a data warehousing appliance typically finds broad support across the enterprise. So why is it then that Teradata migrations seem to fail more often than any other type of database migration? Here are the top 3 reasons why:

1. Project complexity is underestimated due to incomplete knowledge

Accurately estimating the complexity of a database workload borders on the impossible. Yet, being able to quantify the difficulty involved in rewriting a workload is the foundation of any successful migration plan.

Teradata systems are used in demanding multi-purpose, multi-tenant scenarios. They process millions of statements a day. Query complexity ranges from the simple to the sophisticated, from the accidentally complex to the intentionally “clever”. On top of that, Teradata queries frequently use SQL that predates any standardization. These queries rely on the evaluation order of predicates, are case-sensitive in some situations, and frustratingly case-insensitive in others.

It is important to note that the engineers that execute the migration are not to blame here. It really doesn’t matter how much experience a team brings to the table: they will always have to walk into a migration with very little knowledge. They will discover landmines and roadblocks. Simple workarounds will snowball into major rewrites. The complexity of the resulting rewritten statement often eclipses that of the original. All this makes for a challenging cocktail of uncertainty that ultimately translates into inaccurate estimates and impossible timelines and failure.

2. A Teradata migration ain’t easy

Teradata is one of the most powerful database systems. 40 years of maturity might have all but quelled innovation but it is surely a solid database engine. However, over those 4 decades the language surface has considerably expanded, making Teradata also one of the most complex systems on the market. Few systems are equally rich in syntax—or quirks, for that matter. Not surprisingly, this has grave implications for any related migration project.

Take for example Netezza, the data warehouse company IBM acquired a while back. Due to a strict end-of-support policy by the new owners all Netezza instances are undergoing some kind of migration at the time of writing. Netezza’s feature surface is considerably smaller than that of Teradata. Migrating Netezza workloads to a major commercial database systems is therefore a relatively simple task.

In contrast, moving Teradata applications to any other system comes with significant challenges. A complex language surface has to be expressed using less powerful elements of the new target system. Substantial amounts of code need to be added to work around that lack of Teradata syntax and mismatch in semantics. What was originally 30 lines in Teradata SQL may well turn into hundreds of lines in a different SQL dialect. The complexity of the project has just shot up by an order of magnitude. Tempting as it may sound, extrapolating from a much simpler migration to plan a Teradata project is dangerous and usually wrong.

3. Migration is overloaded with modernization

Migrating out of Teradata is often viewed as a rare opportunity to make changes to data model and query statements. Every department has their laundry list of pent-up change requests. Be it data hygiene and janitorial tasks or fundamental changes to how the business works.

What then happens during a migration is essentially a conflation of the actual migration with additional modernization tasks. It seems to make perfect sense at first: if the schema and potentially millions of SQL statements need to be rewritten anyway, might as well make some over-due changes. Who knows when the next opportunity will present itself? Also, delaying modernization and rewriting millions of lines of SQL later again will be frighteningly costly. However, this line of thinking is highly fraught with risk.

Not surprisingly, what started as a project with a clear focus suddenly became “boiling the ocean”. The scope of the project just keeps growing as it drags on. The cost a well-intended modernization imposes on the original migration quickly outweighs the benefits and brings the entire project to a screeching halt. In the end neither party—the migration team nor the modernization team—will be successful.

Here’s what you can do about it

In order for a replatforming approach to be successful it needs to meet 3 fundamental requirements:

  1. Provide complete knowledge about the complexity and difficulty of the task at hand
  2. Establish functional compatibility with existing features
  3. Separate data warehouse migration from subsequent modernization projects

Anything short of disciplined attention to these three is likely to fail. It isn’t an absolute guarantee for failure, but it puts the win probability in the low tens or even single digit percentage range. These are odds, most of us would find completely unacceptable in other areas of daily life under any circumstance.

At Datometry we’ve developed a radically different approach to replatforming that is quite simple in principle and address the three requirements above:

  1. Starting with an in-depth analysis, we simulate the execution of the entire workload for the target cloud data warehouse first. This gives you not only a 360-degree view of the current situation but also peace of mind that there are no bad surprises lurking down the road.
  2. Instead of requiring business applications to be modified, we make vendor-specific features and APIs interoperable—at run time. This leaves both applications and database perfectly intact and unchanged. Think of it as a virtual machine for database applications to run in.
  3. By moving your applications to the cloud this way, you completely separate replatforming and modernization. Enterprise are able to keep the business logic—and yet, move their data warehouse to the cloud. Any modernization can be executed after the move without risking derailing the replatforming effort.

Becoming cloud-native just got a whole lot simpler. Contact us today to stop the guesswork and get results.

About Mike Waas CEO

Mike Waas founded Datometry with the vision of redefining enterprise data management. In the past, Mike held key engineering positions at Microsoft, Amazon, Greenplum, EMC, and Pivotal. He earned an M.S. in Computer Science from the University of Passau, Germany, and a Ph.D. in Computer Science from the University of Amsterdam, The Netherlands. Mike has co-authored over 35 peer-reviewed publications and has 20+ patents on data management to his name.