General application considerations
These are the common considerations that reviewers take into account when reviewing Open Call applications.
When reviewing applications, the following general considerations apply. By keeping these in mind for each application, reviewers help to ensure the Open Call process is consistent, fair, and thorough for all applicants. And, by asking the same questions of each application, Reset is able to best maximize the efficiency and effectiveness of each ultimate award investment.
The following types of considerations (based on areas where we have seen previous projects struggle after they are already underway) are addressed below:
Note: The review considerations provided on this page are the same ones used by the Reset team. As a result, some of these questions/considerations may be less applicable for NIC members. Nonetheless, when evaluating an application reviewers should always include any information they think is relevant regardless of whether it aligns with a specific question/consideration.
These considerations speak to the proposed project's ability to contribute to Reset's Goals and Principles in a relevant manner in one of Reset's core Areas of work.
- Where is the organisation physically and legally based?
- If the organisation is distributed, where is the main point of contact?
- Does the organisation have any conflicts of interest with Reset, the Advisory Council, the Network Investment Council, or other supported projects?
- Is the project team an organisation, community, or an individual?
- Is the organisation or its members already known to our network or within any relevant communities? If yes, what is their reputation and why?
- What are the applicant and/or organisation’s motivations?
- What are the applicant and/or organisation’s principles?
- What are the goals and objectives of the project?
- Is it a technology research, development, or deployment project?
- Can the project’s effort be explained to external audiences and non-technical people?
- What tools, if any, currently exist to solve this problem? How is this project different?
- What problem is the project trying to solve, and is the solution strategical or tactical?
- Is the project strategically or tactically important to our goals, principles, and rationale? To other supported efforts? How?
- Does the effort have any overlap with existing Reset, Luminate, or larger Omidyar Group supported projects?
- Is the overlap complementary or duplicative?
- What is complementary and can it be explained clearly (i.e., geographic focus, technology, organisation profile, etc.)?
- What are the liabilities and risks of taking on this project (i.e., political personalities, financial concerns, technical controversial, etc.)?
- How does the project define its “users”?
- Is the project’s effort actually relevant to its defined users?
- Does the project’s defined user base align with the priorities of Reset?
- Could other priority user groups benefit from this project?
- Are there any existing users? If not, why?
- How do existing users find the user experience? Is it versatile and usable, or useful but unusable?
- What steps has the project taken to appeal to its defined users?
- Does the project have a plan to attract more users and make the project accessible (e.g., conduct usability testing or hold training sessions)?
- What is the user threat model? Who are they being protected from and what are the consequences of failure?
- How is the project localised?
- How is the project representative of those it intends to help? Is it appropriate? What experience does the project have with those it intends to help?
- Does the project measure success quantitatively or qualitatively?
- If neither, how would the project demonstrate success (i.e., does the project have potential quantitative or qualitative metrics)?
- Is there a roadmap for the project? Is it public/open? If not, why?
- Does Reset have the capacity to comprehend and manage the project?
These considerations speak to the proposed project's technical feasibility, capacity, and assets.
- Does the project clearly articulate the technical problem, solution, and approach?
- How is the problem clearly justifiable?
- Does the project clearly articulate its technological objectives?
- Is it an open or closed development project (i.e., open source like Android or Firefox OS, or closed like iOS)?
- Does a similar technical solution already exist? If so, what are the differentiating factors?
- Does the effort propose to sustain an existing technical approach? If so, is the relevant existing approach considered successful?
- Is the effort a new technical approach or improvement to an existing solution? If so, how?
- Is the effort a completely new technical approach fostering new solutions in the field?
- Does the project’s technical approach solve the problem?
- What are the limitations of the project’s technical approach and solution?
- What are the unintended or illicit uses and consequences of this technology?
- Has the project identified and/or developed any safeguards for these consequences?
- How many technologists are actively supporting the project?
- How many of the original technologists are still with the project?
- Is the current team sufficient to meet the project objectives?
- If more people are needed, how does the applicant plan on obtaining the additional technical expertise?
- What other responsibilities and commitments do the project technologists have?
- Is there any existing project code or technical assets?
- Is the existing code or technical assets proof of concept, academic, or production quality?
- How does this technical approach and solution contribute to the larger technical community? How is it measured?
- How does the project make the technical development process visible?
- How is the technical development progress measured by the project?
These considerations speak to the proposed project's financial feasibility in both the short and long term.
- Is the project receiving any other financial support? How is this information disclosed?
- Can we discuss the project with existing financial supporters?
- What in-kind support or other revenue streams is the project receiving (i.e., volunteer developers, service or product sales, etc.)?
- How does the project plan to support itself in the future?
- What is the plan to sustain the project in the future? Is it reasonable and realistic?
- If Reset doesn’t support the project, will it still be realised?
- Does the project provide a detailed and realistic description of effort and schedule (i.e., is the project capable of creating a work plan including objectives, activities, and deliverables)?
- Is the asking amount reasonable and justified?
These final considerations speak to the proposed project's overall qualifications.
In terms of general review considerations, reviewers should base their assessment solely on the merits of the specific project rather than comparing it to other projects for the purposes of resource allocation. While reviewing an application, it is useful to keep in mind the following five questions:
- Are the project's objectives and timeline realistic?
- Should additional collaboration be included?
- Are the proposed outputs tailored to improve the likelihood of third-party use such as utilising more digestible and short form formats?
- Are there clear means of assessing the success of the project?
- What changes or additions need to be made to the proposed effort?
In addition, maintaining a general awareness of these overall considerations can help streamline the review process:
- The proposed project's ability to contribute strategic value to the overall goals of Reset and any applicable host/partner organisations;
- The applicant's qualifications, demonstrated skills, and experience relevant to the proposed project;
- The proposed project’s well-defined goals, outlined objectives, and activities;
- The proposed project’s fit with the host organisation’s mission and area of work (if applicable);
- The host organisation’s suitability, qualifications, and ability to support the project (if applicable); and
- The outputs of the project and the contribution to the larger community in the form of publicly open-sourced code, creative commons licensed content, open hardware licenses, or other models that promotes universal free access to project outputs.
Ultimately, when completing a review, the following twelve questions must be answered in the Evaluation worksheet:
- 1.How clear is the general narrative and strategy?
- 2.How clearly stated are the goals, objectives, tasks, and/or outcomes?
- 3.How well aligned are the described methods and strategy to stated goals and objectives?
- 4.How human-centered or demand-centered is the stated work?
- 5.How feasible are the stated outcomes/how likely are they to be accomplished?
- 6.How well does the application present an adversarial risk analysis?
- 7.How sustainable is this project beyond our support?
- 8.How open and collaborative is the stated approach?
- 9.How realistic and commensurate is the budget with stated goals, objectives, and time-frame?
- 10.How clearly identified is the person or team/how clearly identified are their qualifications to accomplish the work?
- 11.How measurable are the outcomes and milestones/how clear will success or failure be?
- 12.How well does the project align with our own goals and objectives?
Note: The overall length of reviews varies considerably. Please know that we do not have a minimum or maximum expectation of review length for each application/reviewer. Some NIC members may choose to provide the minimum required to submit a review, a sentence or two in answering a question, whereas others may include multiple paragraphs. All are welcome and appreciated.
Last modified 2yr ago