Best way to cross matching large datasets

Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

Your Answer:

Login to answer

Other forums

Wierd echo error?
Hi, i got the most wierd php error ever and i don't know why..
Code: echo "<t

Schedule editor
Hi,

I have a question related to Schedule editor.
If we schedule a BLS for every 30

image upload script not working with png
The basic process of this php code is to take the submitted file, convert it to a small JPG thumbnai

Preventing blank or already entered values
Im trying to learn how to place a read file line by line and prevent duplicate entries and also BLAN

IP question
ive got 2 ip addresses both global from same user how would i detect if they are local to each other

How to use Ajax to verify data on a DB ?
Hello everyone, well i want to know how to check a value if it exists or not on a Data base and capt

help with database debug pls
Parse error: syntax error, unexpected T_ELSE in /home/content/m/e/s/mesick/html/students/05/php/chec

Comparing two dates
Hi Guys
I have two dates I want to compare. Below is my current implementation. The problem with

Simple PHP/Java Help..
Hi,
I'm trying to build a little program to do with calculating postage. Unfortunately i haven't

modifying query string with no page refresh
Hi everybody,
I've tried to get info about my problem all over the internet but i didn't find out