Consultation cum workshop on Strategies for Evaluating SDGs – equity and equality
9th March 2016
National Institute of Labor Economics Research & Development (NILERD) under
NITI Aayog, Government of India in association with the Institute of Social Studies
Trust (ISST) and the Evaluation Community of India (ECOI) organized a consultation
cum workshop on various issues relating to SDGs and gender and equity focused evaluation
in context of India on March 9, 2016 at NILERD, Narela Institutional Area,
The themes on which consultation was convened are as follow. These are the issues
that are being discussed and debated internationally at various forums.
1. Relevance of “new metrics” (measurement tools and indicators) for the evaluation
of SDGs from an equity-focused & gender-responsive perspective
2. Evaluation & complexity – Dealing with the increasing complexity of development
& interconnectedness of SDGs to ensure that “no one is left behind”
3. Towards equity-focused & gender-responsive national evaluation systems –
Multi-stakeholder partnerships to strengthen national evaluation capacities
4. Demand for & use of evidence from equity-focused and gender responsive evaluation
to inform equitable development
The consultation cum workshop was represented from various types of national, international
organizations including Government, academia, researchers and independent evaluators,
NGOs, UN agencies etc.. List of participants is annexed.
A summary of deliberations:
The consultation cum workshop was declared open by traditional lighting of lamp
by Dr. Pitam Singh (retired as Joint Advisor from NITI Aayog) who was joined by
guests and other participants. Dr. Rashmi Agrawal, Director NILERD and core member
of ECOI welcomed all the participants and apprised them about the purpose of the
workshop. Dr. Rajib Nandi the co host discussed the background and current focus
areas of functioning of ISST. He also spoke about the newly formed Evaluation Community
of India (ECOI) and its objectives. Ms. Rituu Nanda elaborated on the process of
the formation and purpose of ECOI. Information on recent developments, including
formulation of action groups to generate knowledge products on various themes, was
shared by Dr. Rashmi Agrawal. Mr. Prasun Kumar from KODE System then launched the
website of ECOI (www.ecoionline.com) and
took the participants through its features.
Dr. Pitam Singh while inaugurating the workshop discussed the structure and mandate
of NITI Aayog ( Earlier National Planning Commission) and stated that NITI Aayog
is placing lot of emphasis on monitoring and evaluation of development programmes.
It is also in the process of establishing regional centers for M&E. The inaugural
session of the workshop was followed by presentations by Mr. Manas Bhattacharyya
representing Association for Stimulating Know How (ASK), Ms. Subhalakshmi Nandi
representing UN Women, Dr. Paramita Majumdar, Ministry of Women and Child Development
added government view point on various issues.
Manas spoke on the role of participatory evaluation in achievement of the SDGs by
ensuring independent assessment of critical aspects of relevance, effectiveness,
efficiency, results, impacts, sustainability, and community participation in specific
development projects & programs; by drawing learning, both in terms
of strengths and weaknesses, and by helping and supporting organizations to develop,
change, modify and strengthen their strategies, approaches, methodologies, nature
of interventions, staffing & mechanisms; by fostering an environment of enhanced
ownership among stakeholders and the community; by integrating “Gender Inclusion”
and “Equity” as a cross cutting aspect/theme in the development programs;
and by focus on participation and utilization taking inclusion and equity
as a cross cutting theme. A need was expressed to maintain cultural sensitivity
while dealing with different communities, without any compromise with the objectivity.
Evaluations need to be empathetic towards the implementing organization and its
staff with the ability to understand their situations, problems, constraints &
challenges while simultaneously not losing the objectivity to be truly participatory.
The process of participatory evaluation should not be treated as a fault finding
exercise rather a process of enabling an organization to overcome constraints which
pose as hurdles in achieving the desired development results. Encouraging continuous
engagement, reflections & open discussions can lead to ownership use-focused
Subhalakshmi explained the role of UN Women which is coordinating across ministries,
across different organizations and key stakeholders, to contribute to global norm
setting and contextualizing and localizing norms. Goals of SDGs cannot be achieved
without achieving the goal 5 of SDGs dealing with gender equality and empowerment
of women and girls. UN is committed to develop a roadmap for women’s status. It
was mentioned that the SDGs are ambitious and transformative. There is a need to
develop a road map for survey on country specific status of women – the existing
state of data and the gaps therein. In particular, a need was expressed to have
data on women’s asset ownership, unpaid work and their time use and crime against
women. Data on gender in the context of evaluation of SDGs is also required.
Paramita mentioned that the principles of gender budgeting has been accepted
in India and is in vogue for over ten years. However, data are collected mechanically
at present and more work needs to be done at the level of development of indicators.
While the financial audits conducted by Comptroller and Auditor General have sanctity
attached to them and are taken due note of, the same is not true of social audit
with gender perspective. Institutional mechanisms are required to give teeth to
social Audits. At present, only some programs are picked up for performance audit.
Internal audit division of Ministry of Finance has been taken on board for mainstreaming
gender. Existing data is weak and various sources provide different data on the
same aspects. For example, Mid -day Meal Scheme was evaluated 42 times by various
agencies with contradictory data and reporting. It is to be seen as to how to reconcile
with various sets of data. The National Policy on Empowerment of Women, focusing
on addressing more gender sensitive issues, is under preparation. Dr. Majumdar expressed
the need to provide inputs in finalizing the policy. Ministries should also come
forward to prepare a checklist for gender guidelines where inputs from various stakeholders
Rashmi indicated that SDGs are not in silos and government has mapped nodal Ministries,
schemes and interventions to be made for each of the SDG targets. She also stated
that mere government formulations would not go a long way; attitudinal changes at
all levels in all sections of the communities can accelerate the process of reaching
gender equality and social equity. Gender equality should not be mystified and problems
of both men and women should be addressed.
Dr. Sunita Palita commented that it is a cause of concern that girls disappear suddenly
from education front after certain years of schooling. They are abused from both
sides, for not being able to do household chores because of school attendance and
in the school not performing well because of domestic tasks. Data are required to
understand this sudden disappearance of girl students.
Ms. Savina Claudia Ammassari mentioned that a lot of work has been done on studying
the vulnerable population. Attention should also be paid to assess outcomes for
India's Third Gender (transgender people).We have to monitor and evaluate outcomes
for vulnerable and often marginalised people, Transegender and other vulnerable
populations who are at high risk of HIV and suffer from significant stigma and discrimination.
TG, sex worker, Homosexuals, drug addicts - all are important groups to keep in
mind and involve in design, implementation and use of evaluations. The most
important part of the evaluation is the USE of data and strategic information to
accelerate progress towards achievement of the SDGs in India and beyond. It is found
that there is a lack of connect between the scientific community and the community
they are serving, that is, the vulnerable population. There needs to
be prioritization of data to change the mindset, to educate the vulnerable, and
to disseminate information. The government needs to be brought on board, and better
coordination needs to happen between the researchers and the decision makers.
Rituu added that if we are looking at power relations, let us not forget those with
other sexual identities as said by Savina. In April 2014, The Supreme
Court of India passed a judgement on issues of transgender, giving them the
right to choose their gender identity and a whole range of development measures
including status under OBC. A recent study in Punjab indicates that school
dropout rate is high in transgender children.
Following the presentations, Ms. Beryl Leach facilitated group formation and group
presentations on four themes identified to be addressed in the M&E systems to
assess SDGs in the Indian context. The group observations on the various themes
are collated as follows.
Theme 1: The relevance of “new metrics” (measurement tools
and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive
Issues raised and outcome of discussions:
It was mentioned that limitations of current data collection methods include lack
of availability of data in public domain/open data, no checklist or cross checking,
concern with regard to data authenticity/reliability, discrepancy, lack of mainstreaming
of data collection methods into organizational information systems and lack of institutional
learning, lack of harmonization and neutralization of data collected across government
and non-government organizations, time and resource constraints. Observations on
the limitations of indicators produced included the following - sex based indicators
do not allow capture of gender dynamics of respondents, lack of sensitivity to gender
and within group vulnerability, inaccessibility to most vulnerable, not focused
on qualitative & non-economic & non-monetary & non-measurable aspects,
lack of checking beyond project period raising concerns on sustainability,
and lack of participatory design. The most difficult issues in measuring social
equity and gender equality and ensuring their sustainability were identified were
- data dissemination, attitudinal and behavioral changes, challenging essential
practices and norms, need for community based plans and stakeholder ownership, capturing
hard to measure discrimination, the need for a methodology sensitive to cover most
vulnerable among the vulnerable (check at design stage) and need for awareness creation.
The new metrics that the group was aware of and which showed the most promise included
Focus Group Discussions (FGDs) and observation based techniques for field research.
During the discussion it was pointed out that there is need for more innovative
metrics suiting to varying cultures and traditions. Only FGDs and observation techniques
were not enough for capturing the change. It was also mentioned that in general
people do not believe in qualitative methods, Some standardization is needed to
make them acceptable. Data analysis plan needs to be established for evaluation
and synchronization between quant-qual is required. Participatory approaches
are of extreme importance. Capacity development to employ qualitative methods is
essential at various levels.
Theme 2: Evaluation and complexity – Dealing with the increasing
complexity of development and interconnectedness of SDGs to ensure “no one is left
Core issues raised in group presentation: The group noted that the challenges
to cope with complexity in evaluation processes include diversity cutting across
inter-state differences, social (caste, class), cultural, power dynamics implying
an inability to rely on a uniform framework and exclusion of people on multiple
levels such as disability and HIV/AIDS. Secondly, environmental dynamism and constant
change causes problems when it comes to developing a concept/logic model. Thirdly,
differences in stakeholder interests and agendas may not be covered – the stronger
and more vocal stakeholders may try to influence the process and outcomes. Solutions
proposed cut across evaluation design and participation, and are listed as follows:
- Participatory design should incorporate flexibility to proceed in an interactive
way - who needs to be involved at what stage, what methods and tools must be used
(informed from reviews of existing information from multiple sources)
- Evaluation design needs to take into account in-depth understanding of local contexts,
needs to be fully-customized and context specific, cookie-cutter approach should
not be there.
- Multi-layer, multi-dimensional, multi-method approach needed for evaluations specifically
to address the issues of E and E.
- Systems approach which identifies “interconnectedness” across global, national,
and local levels and stakeholder mapping
- Participatory monitoring with different stakeholders involved is particularly important
in evaluations of complex programmes
- Strength-based approach can create a space where different stakeholders can share
openly and value each other’s perspective and learn from each other. Composition
of the evaluation team- sensitive to gender and equity lens, has a good mix of thematic
and evaluation experience
- Evaluators need to take the role of facilitators to create an environment
to learn and for evaluation not to be seen as an audit or fault-finding exercise
- Participatory statistics involving community in collecting quantitative data especially
on sensitive issues
Theme 3: Towards equity-focused and gender-responsive national
evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation
Core issues raised in group presentation: An evaluation system is lacking.
There is need for a nodal agency, which might be NITI Aayog, for evaluation with
a steering group to address the specific sectoral, regional and community needs.
The steering group must comprise of state representatives, community representatives
and other stakeholders to look into their specific requirements. This will
provide guidelines and standardized evaluation criteria. An exhaustive checklist
needs to be prepared. Flexible guidelines need to be developed across evaluations.
Capacity building for gender sensitive and equity focused evaluations is needed.
Peer learning and tours must be integral to capacity building to help learn from
good systems. Embedded cells with expert teams will facilitate technical support
for successful partnership in India to strengthen M&E systems in general, and
equity-focused and gender-responsive systems in particular. Every state in the country
has a statistical unit/Planning Department, M&E system should dove tail with
them. The training in M&E system must be made mandatory. The challenges
are that we need to have evolving National Evaluation Policy which will lay down
standardized guidelines and checklist for cross-cutting sectors.
Theme 4: Demand for and use of evidence from equity-focused
and gender responsive evaluation to inform equitable development
Core issues raised in group presentation: The group took a holistic approach
to discuss the factors affecting the demand for and use of evaluations, factors
for demand and utilization of social equity and gender equality evaluations, and
the types of evaluation presentations than can be proposed to make evaluation more
accessible to stakeholders. It was felt that within government a number of factors
affected demand for evaluation including lack of incentives, complacency, high staff
turnover, lack of capacities, lack of understanding so on and so forth. This could
be dealt with providing performance and career incentives, training to improve performance
of project and replication, standardized guidelines and a less rigid framework.
It was also felt that at the level of the implementing agency, conception of the
program should take into account a credible and capable agency which would be gender
responsive. At the level of the evaluators/researchers, the factors included theory
based, mixed method, multi-disciplinary evaluations, time and budget issues.
At present, not much emphasis is given to formative, mid- term and concurrent evaluations
which can be more useful for utilization purposes. All the schemes and programmes
should have a provision for these evaluations.
There is hardly any communication plan for the evaluation reports nor is any specific
responsibility given to individuals/organisations for disseminating recommendations.
Feedback loops for follow up are also missing which shows that evaluations are conducted
without intent for utilization. Even when some reports are placed on web site for
better outreach, a large community cannot access these due to lack of internet facilities,
illiteracy, lack of understanding due to language etc. The communication of
results and best practices should be aligned with the traditions and understanding
of the community. For example at rural level story- telling, role play , visual
media etc can be useful. The community will then raise the demand for use of recommendations.
Private sector in India is coming in big way in developmental activities. There
should be a demand for quality evaluations for their interventions to serve
as evidence for quality decision making. It is important to involve decision makers
in the scheme of things of M&E which is a difficult task. Sensitisation workshops
should be mandatory for them.
The use of evaluations can be enhanced if stakeholders are involved at various stages
of planning of intervention and evaluation questions are a part of intervention
To summarise, program design itself should be gender and equity focused. A movement
for socio psychological empowerment of the vulnerable is a pre-requisite for development
and evaluation, evaluations need to be internalized and focus should be on behavior
change. Evaluations should be context related beating the stereotypes, most under
privileged to be included and sensitization of government and parliament is needed
which will take place with the community pressure. Gender focus lense should be
a way of working with a collective effort. Evaluations are needed both with micro
and macro focus. Interpretation of evaluations should also be without bias. Outreach
of evaluation results should be enhanced using various communication strategies
translating them in the language and understanding of various users. If we are looking
for inclusiveness there is a need to catch the last person which is very difficult.
Strategy should be evolved for how to reach to the last person.
Workshop on Evaluation Strategies for SDGs with E&E- Some Glimpses-9th March