Evaluation worksheet

The following is the evaluation worksheet our team and NIC members use to evaluate Open Call applications.

Conflict of interest disclosure

In line with our policy, please disclose here if you have any financial interest or fiduciary relationship with the applicant or organisation(s) the applicant is affiliated with – this includes, but is not limited to, the receipt or holding of honoraria; educational grants; membership; employment; consultancies; stock ownership or other equity interests; and expert testimony or patent-licensing arrangements.

Personal or professional relationships with the applicant or organisations the applicant is affiliated with which are non-financial and non-fiduciary in nature do not need to be disclosed. If you are unsure of the nature of your relationship with an application or organisation the applicant is affiliated with, please disclose it here.

Do you have any conflicts to disclose?

If no such relationship exists please leave the field blank or write “not applicable.”

Do you confirm that the information provided is true, complete and accurate to the best of your knowledge?

Required checkbox.

Specific topics

Note: Reviewers respond to each question using a scoring system on a 1-5 scale, with 1 being "poor" or "very weak," 5 being "excellent" or "very strong," and 0 being "non-applicable." All of the below are required. In general, those with higher aggregate scores will progress farther in the workflow.

How clear is the general narrative and strategy?

Considerations: Does the application identify and acknowledge what the challenges will be? Does the application state how it is done today and the limitations? Is it clear and specific who could benefit and what the impact will be if the project is successful? Does the application cite an actual use case? Does the application state how much it will cost and how long will it take?

How clearly stated are the goals, objectives, tasks, and/or outcomes?

Considerations: Are the goals and objectives easy to find and read? Are the goals realistically met by the proposed objectives and tasks?

How well aligned are the described methods and strategy to stated goals and objectives?

Considerations: Does the application propose methods appropriate to the goals and objectives? Does the application effectively apply its stated methods? Does the application suggest modified procedures in response to changing circumstances? Will the work fill a potential need or function that is currently unfilled, be reinventing the wheel or creating a solution in search of a problem? Is the proposed solution viable in the real world?

How human-centered or demand-centered is the stated work?

Considerations: Does the effort demonstrate a high degree of usability/accessibility? Does the application demonstrate external demand (i.e., demand originated from existing people with harms vs. potential, hypothetical users, or would-be patrons)? Does the effort's focus appropriately affect either a small number of high value or at-risk users, or a more general large numbers of users?

How feasible are the stated outcomes/how likely are they to be accomplished?

Considerations: Does the application clearly state the effort's technical, research, policy/advocacy, and/or civic engagement outcomes? Are they articulated technically and succinctly? Does the application explain what is new in the approach and why it will succeed? Does the project buy tactical breathing space for existing problems or push towards convergence by changing the playing field? Does the project identify any expected or past hurdles in achieving its technical goals?

How well does the application present an adversarial risk analysis?

Considerations: Does the application identify or recognise potential incentives to an adversary? Does the proposed work increase or decrease known attack surfaces? Does it create new surfaces? Does the proposal consider potential illicit uses? Does it discuss how an adversary might use the solution to further their own goals? Does the application identify potential unintended consequences? Does the application identify and understand the proposed work's asymmetry? Given an attacker and defender of the effort, does the application explain which role is more advantageous (effort, cost, time, etc.) and why? Does the application explain medium and long‐term strategies from both the attacker and defender point of view? Does the application discuss how one could defeat the effort and identify its deficiencies, or does it presume there are none?

How sustainable is this project beyond our support?

Considerations: What is the plan for future development/implementation? Does the application state if the project has other funding sources? Does the project currently receive any Reset, Luminate, or Omidiyar Group support? Does the application identify any cost sharing or matching for the proposed effort? Will the effort be able to support itself by the requested funding, community sources, or other in-kind or indirect support? Does the project have a diversified funding/support stream (i.e., how dependent would the project be on Reset)?

How open and collaborative is the stated approach?

Considerations: Does or should the effort support a collaborative open source community? How will any created intellectual property be licensed/how is it licensed? Will it be freely available to the public? Will there be usable/extensible API's or easy to manipulate text formats? How does the effort facilitate inter-project collaboration, including: talking with other efforts doing similar things and identifying potential points of overlap; acted/planned modularization code to enable others to reuse? Do the application deliverables assist other Reset projects/goals beyond this particular effort? Do the application deliverables assist other Internet freedom projects/goals beyond this particular project?

How realistic and commensurate is the budget with stated goals, objectives, and time-frame?

Considerations: Is the budget realistic and commensurate with both the project needs and its time frame? Is the project going to require funds immediately upon proposal approval?

How clearly identified is the person or team/how clearly identified are their qualifications to accomplish the work?

Considerations: Does the application clearly identify the team member(s) and their work experience? Does the team posses the skills uniquely qualifying them to complete the proposed scope of work? Does the team have a history of successful work related to the current initiative? Is there a core team (leadership, developers, etc.) dedicated to this project? How directly connected or embedded is the team with communities we care most about?

How measurable are the outcomes and milestones/how clear will success or failure be?

Considerations: Does the project articulate a measurable set of evaluation criteria and milestone metrics against results? Are the metrics both quantitative and qualitative? How difficult will it be to conduct an assessment of success or failure?

How well does the project align with our own goals and objectives?

Considerations: Is the proposed project and its trajectory in-line with our Goals, objectives, & areas of work? Is the proposed project and its trajectory in-line with our Principles & values?

Overall recommendation

Note: Only the first question is a required "yes" or "no." All others questions are voluntary, allowing for any general thoughts you may have after reviewing a proposal. The optional text fields provide space to include comments and questions for areas we do not specifically ask for a rating. Comments and questions posted here will be aggregated back to the the rest of the review board and the applicant.

Do you recommend supporting this application?

(Do you recommend supporting this proposal’s statement of work for the requested amount, Yes or No?)

What, if any, positive aspects would you like to highlight?

(Here you can share any specific items that made you really excited or that you see as positive about this application.)

What, if any, concerns would you like to raise?

(Here you can share any specific items that leave you feeling uneasy or that you feel warrant concern regarding this application.)

What, if any, items must be addressed for this to be considered further?

(Here you can share any specific items that warrant dire warnings, a notice of very serious dangers, and/or potential disasters.)

What, if any, other general comments would you like to share with your recommendation?

(Here you can provide any additional comments not captured in or suitable for the above sections.)

Visibility

Who do you want to be able to see your review?

  • Private: Visible only to staff.

  • Reviewers and Staff: Visible to other reviewers and staff.