Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

mysql UPDATE request not working and driving me crazy!!!
Hi,

I have been stuck on this for ages.

Quite simply I am trying to update my database

Help with setcookie()
Merry Xmas to those on this foruum
Older guy here with some experience but not allot so please be

add text and number image
hi guys ..

ok see i want the user to add their name and pick a number then click get it and i

Problems with strings containing
Hey all,

I have encountered something strange in a script. I am trying to write an XML heade

Content-Disposition: attachment; filename=... not working as i thought it should
taken the following code from the php.net site the script is not working.

what is not happeni

Oracle Text CTX_DOC.snippet slow
I have a table (FILE_TABLE) that contains a blob column (ft_file) and I have created the following O

Changes to hosting company's server has affected web contact form
I started a thread a couple of months ago about the contact forms on my client's website. Basically

How can use this array as a key?
Say I have this array for example:


print_r($array);

Prints:
Code: [Select]Array

TabControl Inside ListView Not Editing
Hello everybodyI have a ListView on my page. Inside the ListView I put a TabControl, cuz I want to p

Problems generating word documents on server side for security reasons
I have a problem with word documentation generation when generating a word document (docx) with PHP.