I've defined following tables in my application to fetch the report for every district according to the training date.
wi_individual_g(ind_id, ind_district_id, ...)
wi_individual_p(ind_id,prg_id, ind_dalit (yes/no), ind_madhesi (yes/no), ...)
wi_training(trn_id, trn_start_date, trn_ben_type, ...)
wi_indv_training(trn_id, ind_id)
wi_district(dst_id,dst_name)
My problem: The report has to be generated to count individuals district-wise who are associated with the training between the given trn_start_date. The application has predefined date range with the quarters as defined below:
$quarter=array('y1q3'=>array('2013-02-01','2013-03-31'),'y1q4'=>array('2013-04-01','2013-06-30')
,'y2q1'=>array('2013-07-01','2013-09-30'),'y2q2'=>array('2013-10-01','2013-012-31'),'y2q3'=>array('2014-01-01','2014-03-31'),'y2q4'=>array('2014-04-01','2014-06-30')
,'y3q1'=>array('2014-07-01','2014-09-30'),'y3q2'=>array('2014-10-01','2014-012-31'),'y3q3'=>array('2015-01-01','2015-03-31'),'y3q4'=>array('2015-04-01','2015-06-30')
,'y4q1'=>array('2015-07-01','2015-09-30'),'y4q2'=>array('2015-10-01','2015-012-31'),'y4q3'=>array('2016-01-01','2016-03-31'),'y4q4'=>array('2016-04-01','2016-06-30')
,'y5q1'=>array('2016-07-01','2016-09-30'),'y5q2'=>array('2016-10-01','2016-012-31'),'y5q3'=>array('2017-01-01','2017-03-31'),'y5q4'=>array('2017-04-01','2017-06-30')
,'y6q1'=>array('2017-07-01','2017-09-30'),'y6q2'=>array('2017-10-01','2017-012-31'),'y6q3'=>array('2018-01-01','2018-03-31'),'y6q4'=>array('2018-04-01','2018-06-30'));
If the trn_start_date is choosen as Y4Q4 then, the query must count individuals district-wise for each date range as: Y1(Q1-Q4), Y2(Q2-Q4), Y3(Q1-Q4), Y4(Q1-Q4) separately with single query as:
Y1 Y2 Y3 Y4 Y5 Y6
8 3948 3511 0 0 0
AS a solution, I applied following query:
SELECT wi_district.dst_name,
COUNT(DISTINCT(CASE WHEN wi_training.trn_start_date BETWEEN '2017-07-01' AND '2018-06-30' AND
ind_dalit='yes' THEN wi_individual_g.ind_id END)) AS y6 ,
COUNT(DISTINCT(CASE WHEN wi_training.trn_start_date BETWEEN '2016-07-01' AND '2017-06-30' AND ind_dalit='yes' THEN wi_individual_g.ind_id END)) AS y5 ,
COUNT(DISTINCT(CASE WHEN wi_training.trn_start_date BETWEEN '2015-07-01' AND '2016-06-30' AND ind_dalit='yes' THEN wi_individual_g.ind_id END)) AS y4 ,
COUNT(DISTINCT(CASE WHEN wi_training.trn_start_date BETWEEN '2014-07-01' AND '2015-06-30' AND ind_dalit='yes' THEN wi_individual_g.ind_id END)) AS y3 ,
COUNT(DISTINCT(CASE WHEN wi_training.trn_start_date BETWEEN '2013-07-01' AND '2014-06-30' AND ind_dalit='yes' THEN wi_individual_g.ind_id END)) AS y2 ,
COUNT(DISTINCT(CASE WHEN wi_training.trn_start_date BETWEEN '2013-02-01' AND '2013-06-30' AND ind_dalit='yes' THEN wi_individual_g.ind_id END)) AS y1
FROM wi_individual_g
INNER JOIN wi_individual_p ON wi_individual_p.ind_id=wi_individual_g.ind_id AND wi_individual_g.ind_is_recepient='yes'
INNER JOIN wi_district ON wi_district.dst_id=wi_individual_g.ind_district_id AND wi_individual_g.ind_deleted=0
INNER JOIN wi_indv_training ON wi_indv_training.ind_id=wi_individual_g.ind_id AND wi_indv_training.is_deleted=0
INNER JOIN wi_training ON wi_training.trn_id=wi_indv_training.trn_id AND wi_training.trn_deleted=0 AND wi_training.trn_beneficiary_type=2 AND wi_training.trn_start_date <='2018-06-30'
GROUP BY wi_district.dst_name
But this query takes more than 5 minutes to execute and that's worst. I also applied the index on fields but achieved somewhat same result. I would be thankful if someone provide me the best solution.
I've found the way to increase the performance by 3-times:
At first : the query took around 128 secs
After suggestion: the query took around 78 secs
Further modification: the query took around 23 secs
---------------------------------------------------------------------------------
SELECT d.dst_name,
COUNT(DISTINCT(CASE WHEN a.trn_start_date BETWEEN '2014-07-01' AND '2015-06-30' THEN a.ind_id END)) AS y3 ,
COUNT(DISTINCT(CASE WHEN a.trn_start_date BETWEEN '2013-07-01' AND '2014-06-30' THEN a.ind_id END)) AS y2 ,
COUNT(DISTINCT(CASE WHEN a.trn_start_date BETWEEN '2013-02-01' AND '2013-06-30' THEN a.ind_id END)) AS y1
FROM
(
SELECT g.ind_district_id,g.ind_id,t.trn_start_date,t.trn_beneficiary_type
FROM wi_individual_g g
INNER JOIN wi_indv_training wit ON g.ind_id = wit.ind_id AND wit.is_deleted = 0 AND g.ind_deleted=0 AND g.ind_is_recepient='yes'
INNER JOIN wi_training t ON wit.trn_id = t.trn_id AND t.trn_beneficiary_type=2 AND t.trn_deleted = 0
) a
INNER JOIN wi_individual_p p ON p.ind_id=a.ind_id
INNER JOIN wi_district d ON d.dst_id=a.ind_district_id
WHERE p.ind_dalit='yes'
GROUP BY d.dst_name;
As a whole, the performance has been increased by 6-times from my previous query. Thank you for the suggestion @DRapp
If anyone having best solution to increase the performance, I would like to thank him!
I have slightly altered the query to put adjust the criteria to the respective join, or into the WHERE clause where applicable. I also moved the "ind_dalit=yes" component to the JOIN to the wi_individual_p table vs in each case statement.
With this, I can better see criteria to offer suggestions of indexes including
table index
wi_individual_g ( ind_is_recipient, ind_deleted, ind_id, ind_district_id )
wi_individual_p ( ind_id, ind_dalit )
wi_district ( dst_id, dst_name )
wi_indv_training ( ind_id, is_deleted )
wi_training ( trn_beneficiary_type, trn_deleted, trn_start_date, trn_id )
SELECT
d.dst_name,
COUNT( DISTINCT( CASE WHEN t.trn_start_date
BETWEEN '2017-07-01' AND '2018-06-30'
THEN g.ind_id END)) AS y6,
COUNT( DISTINCT( CASE WHEN t.trn_start_date
BETWEEN '2016-07-01' AND '2017-06-30'
THEN g.ind_id END)) AS y5,
COUNT( DISTINCT( CASE WHEN t.trn_start_date
BETWEEN '2015-07-01' AND '2016-06-30'
THEN g.ind_id END)) AS y4,
COUNT( DISTINCT( CASE WHEN t.trn_start_date
BETWEEN '2014-07-01' AND '2015-06-30'
THEN g.ind_id END)) AS y3,
COUNT( DISTINCT( CASE WHEN t.trn_start_date
BETWEEN '2013-07-01' AND '2014-06-30'
THEN g.ind_id END)) AS y2,
COUNT( DISTINCT( CASE WHEN t.trn_start_date
BETWEEN '2013-02-01' AND '2013-06-30'
THEN g.ind_id END)) AS y1
FROM
wi_individual_g g
INNER JOIN wi_individual_p p
ON g.ind_id = p.ind_id
AND p.ind_dalit='yes'
INNER JOIN wi_district d
ON g.ind_district_id = d.dst_id
INNER JOIN wi_indv_training wit
ON g.ind_id = wit.ind_id
AND wit.is_deleted = 0
INNER JOIN wi_training t
ON wit.trn_id = t.trn_id
AND t.trn_beneficiary_type = 2
AND t.trn_deleted = 0
AND t.trn_start_date >= '2013-02-01'
AND t.trn_start_date <= '2018-06-30'
WHERE
g.ind_is_recepient = 'yes'
AND g.ind_deleted = 0
GROUP BY
d.dst_name
Here is another option you might try. This pre-queries (alias PQ) distinct "g" district and ind_id per date group 1-6 vs returning every dated record. Then the result is a simple sum per district.
SELECT
d.dst_name,
SUM( PQ.DateGrp = 6 ) AS y6,
SUM( PQ.DateGrp = 5 ) AS y5,
SUM( PQ.DateGrp = 4 ) AS y4,
SUM( PQ.DateGrp = 3 ) AS y3,
SUM( PQ.DateGrp = 2 ) AS y2,
SUM( PQ.DateGrp = 1 ) AS y1
FROM
( select distinct
g.ind_district_id,
g.ind_id,
CASE WHEN t.trn_start_date BETWEEN '2017-07-01' AND '2018-06-30' THEN 6
WHEN t.trn_start_date BETWEEN '2016-07-01' AND '2017-06-30' THEN 5
WHEN t.trn_start_date BETWEEN '2015-07-01' AND '2016-06-30' THEN 4
WHEN t.trn_start_date BETWEEN '2014-07-01' AND '2015-06-30' THEN 3
WHEN t.trn_start_date BETWEEN '2013-07-01' AND '2014-06-30' THEN 2
WHEN t.trn_start_date BETWEEN '2013-02-01' AND '2013-06-30' THEN 1
ELSE 0 END DateGrp
from
wi_training t
JOIN wi_indv_training wit
ON t.trn_id = wit.trn_id
AND wit.is_deleted = 0
JOIN wi_individual_g g
g.ind_is_recepient = 'yes'
AND g.ind_deleted = 0
AND wit.ind_id = g.ind_id
INNER JOIN wi_individual_p p
ON g.ind_id = p.ind_id
AND p.ind_dalit='yes'
where
t.trn_beneficiary_type = 2
AND t.trn_deleted = 0
AND t.trn_start_date >= '2013-02-01'
AND t.trn_start_date <= '2018-06-30' ) PQ
INNER JOIN wi_district d
ON PQ.ind_district_id = d.dst_id
GROUP BY
d.dst_name