Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

login to other site by sending post variables
hey,

here is my problem: my school gave me a mail account for school-stuff use. they mail us

Warning: Cannot modify header information - headers already sent by (output sta
Warning: Cannot modify header information - headers already sent by (output started at /home/praylif

LOOPing Problem
Hello All!

The following code loops through the data and displays the data accordingly. My p

Email Form Syantax Issue 2
How to make this look FROM:"OUR TEAM" instead of FROM:abc@mail.com

$headers = "

Alternate messaging
I have 4 strings in MySQL db1

$string1 : Hello
$string2 : Hi
$string3 : Great
$strin

javascript in php code causes page to display white
Hi All,

Odd one this, I've got the following code (below) which is a simple log in form. The

Problem related to Creation of PDF File?
Hi All,
I am facing a problem related to creation pdf file. when I am creating a pdf file of do

Setting/Modify Headers using mail() function
I have a question about setting email headers when using mail() in a php script.

When I send

Wrong Info sent Using $get in form
Hi i have this code that i tried to make to send some info from one page to aother with lots of ppls

Contents of variable not echoing
Hey guys, hopefully this is an easy one...

In this line, the variables are not echoing out. T