Can we improve the current state of benchmarketing?

http://ift.tt/1xOppNM

I'm starting off 2015 with the following New Year's Resolution, to improve the state of benchmarking. About a month ago I noticed the following tweet:Hey @tokutek, please look at this: http://ift.tt/1Du82DT …. Are the benchmarks rigged or correctly done? I'm curious to know!While I've never met Ian Campbell (@iamic) he certainly knew how to call me to action. I immediately checked out the STSsoft website, the benchmark results page, and the benchmark code itself. My first reaction was that something had to be wrong, as the benchmark results showed TokuDB and MyISAM requiring the same amount of disk space. FYI, MyISAM does not compress at all unless it's in a compressed read-only mode.Needless to say it was time to figure out what was really going on with this benchmark. As was the case with the EnterpriseDB NoSQL Benchmark that I reviewed back in October, I decided to dig in, review the benchmarketing, and dissect the benchmark itself.Step 1: Benchmarketing ReviewTo keep my process bounded, I decided to only review the compression claims as stated on the STSdb v4.0 product page. The page also makes serious performance claims versus Fractal Tree indexing technology that I'll likely test and blog in the future. The page clearly states:"Up to 3x more compact than TokuDB."Underneath the claim is a link to the benchmark results page. In the size chart at the bottom of that page it shows STSdb 4.0 at 5365 MB, MyISAM at 7051 MB, and TokuDB at 7051 MB. So I was left wondering...On what planet is 5365 3x smaller than 7051?For the claim to be true the STSdb size would need to be 2350 MB.Why is the MyISAM size exactly the same as the TokuDB size?The answer to the second question causes a serious benchmarketing issue for the vendor. The SQL that the benchmark uses to determine size was generic for both MyISAM and TokuDB. In the TokuDB case it is calculating uncompressed size, which explains why it was the same as MyISAM. This could also have been checked by looking at the size of the files on disk. At the bottom of this blog I've included the benchmark code change required to properly determine the size for TokuDB.Step 2: Run the benchmark (including the fix)Here are my results for MyISAM and TokuDB. I'm including MyISAM results to show that I'm running the benchmark properly (comparing to the posted results).Note that I'm running on TokuDB v7.5.3 for MySQL 5.5.40, stock defaults (no TokuDB variables defined in my.cnf), on an Ubuntu 14.04 desktop with a Core i7-4790K, 32GB RAM, and an Intel 480GB SSD. The benchmark client is running in a Windows 7 Virtual Machine (VMware Workstation 11.0) on the Ubuntu desktop.Step 3: Analyze the resultsTokuDB is 18% smaller than STSdb (4413 MB vs. 5365 MB).Where do we go from here?Can we do better than this? I think we can. I propose the following:Do everything possible to make sure you publish accurate results.If something looks too good to be true, it probably is.Review the benchmark efforts of others.Even if it's not comparing to your technology, peer review is needed.Challenge incorrect results.I welcome others to review my benchmarks and my results as it only makes the benchmark more trustworthy.The correct SQL to determine TokuDB size:try{  string tables = String.Join(" OR ", Enumerable.Range(0, connections.Length).Select(x => String.Format("table_name = '{0}'", GetTableName(x))));  string query = "";  if (StorageEngine == MySQLStorageEngine.TokuDB)  {    query = String.Format("select sum(bt_size_allocated) from information_schema.TokuDB_fractal_tree_info where table_schema='{0}' and ({1});", conn.Database, tables);  }  else  {    query = String.Format("SELECT SUM(Data_length + Index_length) FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = '{0}' and ({1});", conn.Database, tables);  }  IDataReader reader = conn.ExecuteQuery(query);  long size = 0;  if (reader.Read())    size = reader.GetInt64(0);  reader.Close();  return size;}

from Planet MySQL http://ift.tt/1du18ol