Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Conditions of info record - Error
When i created GR (901) and PO create automatic my PBXX is obtain the net price of info record but

MII Netweaver Server stopped due to CPIC connection Problem
Dear Experts,
We have SAP XMII 12.0 setup running on Netweaver 2004s,7.0, Java Stack only.

How to ask a regex question?
Hi, I thought I'd share some of my insight after seeing too many bad questions that either get ignor

phpMailer not working ..........
HI,
I was playing with the phpMailer script and tried to send a smiple mail using their test scri

Struct/union and scope problem!
HI all , I have
Code: in header.h typedef struct Node Link; /* ---------

Is STL important?
I'm just starting programming and I've made it until the Standart Template Library. But the chapter

Converting from one format to another
"I have got a date in DD/MM/YYYY but I need it in MM-DD-YYYY. Help!"

Is there a quick way to do this?
Hi all,

I need to create a mysql table from php that goes a little something like this

Load file in PHP
I have 2 files. The first is a PHP generated XML file that's dependent by 2 inputs. The second is a

Get keys and values from an array
Hellow,

I have an array $Data with keys and values. With a foreach I can display all the valu