Best way to cross matching large datasets
Posted on
16th Feb 2014 07:03 pm by
admin
Hi,
Im running a script where am I cross matching about 200 000 data sets with each other. Each data set consists of 8 parameters and I want to count all datasets which have similar or the same parameters for each data set.
Right now, I am doing the matching via a MySql query which im calling about 200 000 times. The problem is that using a query is extremely expensive… it takes up to 2 hours until the script is done. So I am wondering if there is a better method to cross match data sets and if some of could help me find a better solution.
While researching I found out that arrays may be a faster alternative to queries. And so far, I identified 3 possible ways for cross matching:
1. nested foreach () loops
foreach($array as ar1)
foreach($array as ar2)
if ($ar1[0] == $ar1[0])….
2. Using an Array_map with Callback function, so that i would have only one "hand coded" loop
foreach($array as arr)
if ($arr[0] == $parameter)….
3. Array walk where i could save one "hand coded" loop as well.
Theoretically would be the best/fastest way to go about it? Can Anyone tell me what technically the difference between those 3 ways is? And which one is the better approach or if there other alternatives to them?
I am thankful for any advice that helps me reduce execution time!
No comments posted yet
Your Answer:
Login to answer
343 
48 
Other forums
need help with date function
i have following form for date
Code: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01
Give me all your tricks for minimizing jar file size
Hi, I'm coming close to releasing my J2ME game... I am kicking up against the 64k size barrier w
Price in PR
Dear Experts,
Why we have to enter the price in the PR, and why there is no free goods indicato
Accessing element of object array
Hello
My object looks like this:
Array ( [0] => User Object ( [id] =>
passthru() help
Hi everyone,
I'm new to the forum and to PHP. I have some pretty good experience writing shell sc
Problems with returning true or false in eval()'d code
Hi guys,
Would appreciate some help with a problem when running eval() on a function that sho
Using CSS to format MySQL query
Hi,
If i want to have a news page on my site, which displays all records of a table in descen
strtotime issue
Hey all,
I'm playing around with some code, and basically the idea is:
Person changes
NOOB needs help with upload file
The intent of this form is to create a folder for a client and upload an image into another folder '
please fix the error
What is the error in the below code ???
Line number On/Off | Expand/Contract <?php
