Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

why aint this working?
i have this code:

Code: $num = 250;
$count = ($num - $rcount);

echo $rc

Trouble checking SESSION cookie
I am trying to use $_SESSION cookies to verify admin privileges .
I don't understand why this is

Production Order Enterprise Service to Manufacturing Exctn Sys (MES) ???
Hello Experts,

In our current landscape SAP ECC 5.0 is integrated to MES system via PI 7.

Hyperlink in PHP to MySQL record
Hello,

I am having trouble as i am new to php. I have connected to the database and I have di

Greek characters in php
Hi,

I'm making a script and I m using for first time greek characters.
I started to write

scandir clients directory
hi,
how can i scandir the clients directory? i need a script that when i click a button it will u

how can i display php source code snippets ?
I am outputting some pho code .. and I want to display the source code so people can copy and paste

how can i make a .gif images for my site
Dear Friends ,

Pls solve my problem , I am starting making new site but i dont know how can i

order by date not ID number help php
I am trying to orginize the following code to order by date not id number.
any help would be grea

do while...woes
I'm sure this is childsplay but it is stumping me, everytime i try what I think would make this work