Best way to cross matching large datasets
Posted on
16th Feb 2014 07:03 pm by
admin
Hi,
Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.
Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.
While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:
1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….
2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….
3. Array walk where i could save one "hand coded" loop as well.
Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?
I am thankful for any advice that helps me reduce execution time!
No comments posted yet
Your Answer:
Login to answer
343 
48 
Other forums
Need help-Error istream header declaration
Hi Everyone ,
I am having issue with istream declaration. I am new to C programming.
can a
Help with file_get_html
The following code gives the expected results:
<html>
<head&g
BackButton Behaviour in AJAX
I have an ASPX Page AJAX Enabled!The page has a gridView and a DDL for filtering on it. The Gridview
Shuffle between users ??
I have multiple $users in table. I need to send them $message.
I need to send the next messag
Problem executing bash script using shell_exec
Hi there,
I created a bash script file using following code to convert doc documents to pdf using
how to get values from $_Post with onchange="form.submit();"?
Hi, I have a form that submit to itself with a file upload field and two other hidden field values.<
error reporting
hey everyone
I have a production server in which I want error reporting but only on some page
Remote Database Access
Hey guys!!
OK, i'm used to only working with databases from the CPANEL, accessing phpMyAdmin
How do I get the row number from from an sql table query
Hi all
I have a table that I query and it returns a number of rows.
mysql_num_rows($query)
get font info from a font file
hello,
Does anyone know how to get font info from a font file ... using php of course !
<
