Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

array ...
hi
Code: function formatCategories($categories, $parentId)
{
// $navCat stores all child

Looking for help on using joining mysql tables and php....
Does anyone have any good links to references on how to pull mysql data, through joining tables and

Delete Client 066 Earlywatch
Hi all,

in former times client earlywatch was required for SAPs remote access to SAP inst

How would I do "Users not logged in since *** delete them"
I know how I would set this up, and delete the user roughly.
I'll create a table called "Use

redirect standard error and assert (how to?)
How do you redirect standard error and assert?

---

(Why? I am creating an web app and

I apologise! It's a newbie to PHP feedback form problem!
Hi,

I am a total newbie to PHP. I am also a unemployed and need to get a job. So after 100% r

very easy question about SQL info
Hello,
I think that I have a very easy question.
I know how to create a form using php SQL and

Multipe Dynamic Controls & AutoPostback Issue
I currently have a need to create many dynamic controls (Example Textboxes) that need to do a PostBa

How to get variable value on next page
Hello friends
i am working on payentry page ..there i have this code.........

i need help with php header and footer
I have designed a header and footer for my site and they seem to be ok when they are running individ