Best way to cross matching large datasets
Posted on
16th Feb 2014 07:03 pm by
admin
Hi,
Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.
Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.
While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:
1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….
2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….
3. Array walk where i could save one "hand coded" loop as well.
Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?
I am thankful for any advice that helps me reduce execution time!
No comments posted yet
Your Answer:
Login to answer
343 
48 
Other forums
Coefficient of a Restitution Hints
Hello. I was wondering if anyone can give me hints on how to write this program...
The coeffi
strtotime issue
Hey all,
I'm playing around with some code, and basically the idea is:
Person changes
Problem with PHP code- simple contact form
I'm relativily new to PHP; I know HTML and CSS stuff but I have a problem- I have a contact form wit
phpMailer will not connect using SMTP
I am trying to use phpMailer with smtp:
Code: [Select]$mailer = new PHPMailer();
$mailer-&
Regarding accessing SQL query issued by any user in Oracle 10g
Hi all,
i want to know the queries issued by various users accessing a database...
Revoking alter any table from a schema
Hi All,
i am working in Oracle 10g.
my requirement is to revoke all kind of DDL oper
writing my own sobel filter convolution - something is wrong
I am trying to keep it very simple, I cant see anything wrong with my logic, could anybody help poin
help with mysql_error()
Hi,
I am trying to insert data into a table, but I am not able to insert it. I wanted to see
frame help
on the bottom frame is menu.html
Code: <body><form action='link.php' met
Data Function is Its Not working IN IE8
This is my first time to use formums. I hope i can get solution for this problem.
view plaincopy to
