Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

(sub)screen resize modification doesn't work after ECC 6 EhP 4 installation
Hello,

We have previously extended the size of the custom data tab on PO transaction ME23

mysql query with single quotes in a variable
$sitedetails = "INSERT INTO vars (address, sitename, description, ownername, theme) VALUES ('$u

php sessions,logouts & the bloomin back button!
Hi All,

I've got a cms that members can log into. When they logout, the session is destroyed,

"From field" in PHP email form
I used a wizard to create a PHP email form. I was able to customize it with the exception of the Fr

Validation
Hello

I know this is a really vag question.

I need to have a input field on a form th

Need help making a blockquote and line items conditional
I have some code I bought a few years ago that allows my clients to update content on their site usi

'210010106140040100' == '210010106140040101'
Debugging this simple line of a PHP script

Code: if($a == $b){ }
I've found that with val

Drawing Images in classes
I'm quite new to Object Oriented PHP. What I'm trying to do is draw some images but also have other

Will this protect from mysql injection?
I do not want anything like DROP TABLE to work or any type of coding..

Heres my code.. is it

How to generate a random array of integers subject to a certain constraint
I am writing a program that asks the user to enter an integer N and then generates a random array of