Datometry achieves 99.5% coverage in industry-first

Celebrate Your Database Independence Day

06.30.20

In a recent benchmark, Datometry put out some truly impressive numbers: On average, customers can expect Datometry Hyper-Q to give them near-complete coverage at 99.5%. Datometry customers replatform at a fraction of time, cost, and risk.

Let’s step back for a moment. Hyper-Q is a virtualization platform that lets enterprises run applications, originally written or configured for databases like Teradata, natively on a cloud data warehouse — without extensive changes to SQL or APIs. In other words, Datometry eliminates the painful and highly error-prone process of database migrations.

Anybody who’s ever tried to move from one database to another knows just how excruciating and demoralizing these multi-year migration projects are. Sadly, the vast majority of them fail. And attempts to replatform away from Teradata have a particularly low rate of success.

So, what does the result of 99.5% coverage imply for CIOs of the Fortune 500 and Global 2000? How to put it into perspective? And what exactly does it mean for your upcoming replatforming project?

What data did we analyze?

Together with our cloud partners, we collected 75 Teradata workloads from leading enterprises worldwide. These are complete workloads reflective of what the modern enterprise processes to power the full data life cycle. Also, these firms are in the process of moving off of Teradata entirely and intend to leave absolutely no workload behind.

Now, the workloads we examined are complete recordings for a given period of time. They typically cover a minimum of seven days and up to three months. All of them are replete with (1) ETL from both standard third-party systems and custom scripts, (2) BI and reporting, as well as (3) interactive analytics.

In the context of Teradata, these are workloads that often originate from an array of utilities including MultiLoad, FastLoad, FastExport or TPUMP. Another favorite is BTEQ, which is frequently used to execute sophisticated data modifications. ODBC and JDBC enable a variety of custom software products to interact with the enterprise data warehouse.

In a replatforming project, all of these interactions need to be taken care of. So, completeness is critical for our analysis. And as if that wasn’t enough, we ask our customers for their gnarliest and most demanding workloads: end of quarter reporting, end of year closing of the books, etc. Our test sets have it all.

What did we measure?

We ran all workloads through Datometry qInsight. qInsight processes these workloads by emulating every single request using Hyper-Q, the virtualization platform. All DDL, all DML, all queries, in short, all requests are replayed during this analysis. As part of the processing, every request is synthesized into one or more SQL statement for the destination system as would be in a production setting.

During the analysis, we look for any requests for which Hyper-Q cannot produce bit-identical results. And while a single request might consist of a large combination of individual features, a single unsupported one will lead to a failure in our experiment. It isn’t hard to see that even a single minor feature that is not yet supported can have a significant impact on the overall workload.

For each workload, we measured the total number of requests/statements for which emulation does not succeed. Typically, these are variants — i.e., corner cases — of otherwise well-supported features. They are rarely sophisticated or complex queries. Most often, they pertain to differences in the interpretation of data types. And surprisingly often, they are queries that are simply unintended and incorrect. They would have failed earlier if it weren’t for the permissive nature of the source system. Our customers appreciate it when we catch these.

Once all workloads are processed, we average over all experiment and all workloads. It is this experiment that produced the remarkable — and to most practitioners highly astonishing — result of 99.5%. This accomplishment is the result of several years of research and development by one of the top teams in the industry.

How does this compare to industry standards?

First off, Datometry’s technology is unique. There isn’t any directly competing software platform. Even though database owners have grappled with the problem of moving from one database to another for over four decades. The standard proceeding has long been manual rewrites of the entire body of SQL and the retooling of data pipelines and consumers.

Given the high importance to the overall industry, an entire crop of methodologies has entered the market. They perform and partially automate what we refer to as static translation. That is, a consultant extracts the SQL from an application, runs it through a cross-compiler, adjust it if it doesn’t compile, and reinserts it into the application.

Unfortunately, static translation is limited in its capabilities. It simply cannot deal with sophisticated features ranging from the recursion to Stored Procedures; from global temp tables to case-sensitivity; from macros to the support of the existing client tools. In these cases, it’s on the application owners to modify their code and make up for the discrepancy.

This technique of static translation succeeds in about 70% of the cases. Given that replatforming is an 80/20 problem where the last 20% of the problem requires 80% of the effort, a 70% solution isn’t much of a breakthrough. Rather, automated static translation is a Band-Aid that lulls clients into the sense of rapid progress but will not be able to overcome the hard problems.

At 99.5%, Datometry is not perfect — yet. But it is easy to see Datometry outperforms translation tools by an impressive margin.

Datometry customers replatform at a fraction of time, cost, and risk

99.5% is a pretty strong result. For IT leaders looking to move the enterprise to the cloud, it means:

  • There are practically very few things left that may require manual intervention during replatforming.
  • The broad selection of workloads under our study demonstrates the general applicability of database virtualization with Datometry Hyper-Q.
  • The ability to analyze the entire workload up-front ensures there are no surprises later in the project.

Importantly, this analysis is available to every Datometry customer at the beginning of the project. We know how important clarity is in this case. Often, the replatforming from Teradata — be it from on-premises appliances or public cloud — to a cloud-native data warehouse is the single most important IT project of the current fiscal year.

Not only will this analysis give you clarity on whether your workloads contain elements that Datometry does not yet cover. It gives you exact insight into all risks and a concrete plan of attack to execute. So you can step into the cloud with confidence.

To jump-start your journey to cloud-native data warehousing, contact your public cloud vendor today and ask about their Datometry evaluation offer. You may qualify for an assessment free of charge, a $6,499 value. For additional information, contact us at info@datometry.com.

Replatform to a modern cloud data warehouse in 6 simple steps
Learn More
Author avatar

About Mike Waas

CEO

Mike Waas founded Datometry with the vision of redefining enterprise data management. In the past, Mike held key engineering positions at Microsoft, Amazon, Greenplum, EMC, and Pivotal. He earned an M.S. in Computer Science from the University of Passau, Germany, and a Ph.D. in Computer Science from the University of Amsterdam, The Netherlands. Mike has co-authored over 35 peer-reviewed publications and has 20+ patents on data management to his name.

Continue Reading