Best way to cross matching large datasets


Posted on 16th Feb 2014 07:03 pm by admin

Hi,

Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.

Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.

While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:

1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….

2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….

3. Array walk where i could save one "hand coded" loop as well.

Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?

I am thankful for any advice that helps me reduce execution time!

No comments posted yet

Your Answer:

Login to answer
343 Like 48 Dislike
Previous forums Next forums
Other forums

Need help-Error istream header declaration
Hi Everyone ,

I am having issue with istream declaration. I am new to C programming.
can a

Help with file_get_html
The following code gives the expected results:

<html>
<head&g

BackButton Behaviour in AJAX
I have an ASPX Page AJAX Enabled!The page has a gridView and a DDL for filtering on it. The Gridview

Shuffle between users ??
I have multiple $users in table. I need to send them $message.

I need to send the next messag

Problem executing bash script using shell_exec
Hi there,
I created a bash script file using following code to convert doc documents to pdf using

how to get values from $_Post with onchange="form.submit();"?
Hi, I have a form that submit to itself with a file upload field and two other hidden field values.<

error reporting
hey everyone

I have a production server in which I want error reporting but only on some page

Remote Database Access
Hey guys!!

OK, i'm used to only working with databases from the CPANEL, accessing phpMyAdmin

How do I get the row number from from an sql table query
Hi all

I have a table that I query and it returns a number of rows.
mysql_num_rows($query)

get font info from a font file
hello,

Does anyone know how to get font info from a font file ... using php of course !
<

Sign up to write
Sign up now if you have flare of writing..
Login   |   Register
Follow Us
Indyaspeak @ Facebook Indyaspeak @ Twitter Indyaspeak @ Pinterest RSS



Play Free Quiz and Win Cash