Creating your first evaluation

Great - so you've logged into Culture Counts, but you don't know where to start. If you're having trouble getting setup, the article below will take you through the steps required to create a new evaluation, and making your first survey!

When you log in to your Culture Counts account, you will see that we have provided an example evaluation based on a set of standard metrics that have already been benchmarked internationally. You can clone, then edit, individual surveys from this evaluation in order to get a jump start, or you can create your very own evaluation. We recommend first time users follow the example evaluation, but you should try both and see what suits you best!

In this article:

 

What is an evaluation?

Evaluations are folders or containers that contain all the information for a particular event, project, or place that you wish to evaluate. If you wish to create several related surveys to evaluate an event, they can all be grouped and organised within a single evaluation. The evaluation will store and present all the data collected through those surveys so that you can quickly see what an event is achieving overall. In time the evaluation will also store and manage any data about the event that you have imported from elsewhere.

 

Creating an evaluation by cloning the Example Evaluation

About the Example Evaluation

We have created an Example Evaluation to give you some insight into how we design our evaluations.

There are three types of surveys that can be created within an evaluation: a ‘standard’ survey, a ‘prior’ survey and a ‘post’ survey.

The most basic form of Culture Counts evaluation contains three standard surveys:

  • a standard objective-setting survey for self-assessors which asks your self-assessors to set out the scores they hope to achieve for the event in relation to particular dimensions of quality
  • a standard survey for peers which asks them to rate their experience of the event against the same quality dimensions
  • a standard survey for members of the public to rate their experience of the event, again based on the same quality dimensions

For more advanced forms of evaluation you may wish to ask your peers – and perhaps even your self-assessors – to complete both a prior and post survey, rather than a single standard survey:

  • a prior survey asks respondents to describe their expectations of the event; afterwards you can compare expectations with how the event was actually perceived
  • a post survey asks respondents about their actual experience of the event; when linked to a prior survey, it allows you to measure the difference between expectations and reality

For more information on standard, prior and post survey types, and which survey types to use for different assessors, see ‘Configuring your survey’.

Our Example Evaluation contains examples of three standard surveys (self, peer and public) and one prior survey and one post survey (self and/or peer). They are based on a set of tried-and-tested, standardised metrics. We suggest you start by browsing through the surveys in the Example Evaluation to see how they have been set up. 

Please note that the Example Evaluation is a shared resource that is available to all Culture Counts users. We therefore ask you not to make or save any changes to the surveys within the Example Evaluation itself, as any changes you make will also show up in the Example Evaluation in the accounts of other users! Please clone the surveys first before you start editing them.

Cloning the Example Evaluation

To create a new evaluation by cloning the Example Evaluation, you first need to create a new evaluation for your event. This will be a new container where the cloned surveys can be stored. To start a new evaluation, click the New evaluation button at the bottom of the Evaluations panel. You will be prompted to give your evaluation a name (normally the name of the event you are evaluating) before being taken to the evaluation Overview screen.

To clone the surveys in the Example Evaluation, first click on Example Evaluation so that you are presented with the example surveys described above. We suggest you start by cloning the three standard surveys for self, peer and public assessors. To clone the example public survey, click Edit and select Clone from the dropdown list. Give the cloned survey a new name and choose the evaluation you just created to clone it into.

We recommend giving each survey a simple name that both people in your organisation and assessors will understand, like the name of the event and the survey type: ‘My Awesome Event – Public Survey’ or ‘My Awesome Event – Peer Prior Survey’.

Your new evaluation will contain all the surveys that you have just cloned from the Example Evaluation. You can now edit these surveys however you wish. Click on your new evaluation and on the Overview screen click on the name of the survey that you wish to edit. You will be taken to the survey Summary page, which shows a brief summary of the survey settings and questions included. From here you can edit the survey using the three options in the drop-down list that appears when you click on the Edit button:

  • Configure – to change the survey settings
  • Design – to edit, add or remove survey questions
  • Invite – to add or remove self and peer assessors

 

Creating an evaluation from scratch

To start a new evaluation, click the New Evaluation button at the bottom of the evaluations panel. You will be prompted to name your evaluation (normally the name of the event or place you are evaluating), before being taken to the evaluation's overview page. From here you can create new surveys, manage existing surveys and download any data that has been collected for the evaluation.

To create a survey for your new evaluation, click Create survey. You will then be taken through the three main stages of survey creation:

  • Configure – to change the survey settings
  • Design – to edit, add or remove survey questions
  • Invite – to add or remove self and peer assessors

 

Configuring your survey

The Configure page is where you set up the following options for your survey:

Name

This is the survey name that respondents will see when they access the survey.

Start and close dates

The start and close dates will determine when the survey begins and stops taking entries. If you leave the start date blank, the survey will begin immediately, and if you leave the close date empty the survey will keep the survey active indefinitely.

Survey type

You can choose one of three survey types for your survey:

  • Standard: for general surveys that will be taken once by respondents. Choose this type if the survey is not designed to measure differences between prior expectations and experience
  • Prior: for surveys where you want to measure the expectations of an event, so that they can be compared later with how the event is actually perceived
  • Post: for surveys where you want to measure how an event was rated and compare this with the prior expectations of the same respondents

If you wish to include an assessment of the difference between expectations and reality in your evaluation, you need to create a prior and a post survey and link the two together. The easiest way to do this is to create the prior survey first. When you create the post survey, select the correct prior survey from the dropdown list under Survey Link on the Configure screen.

Delivery types

This section allows you to choose how the survey will be delivered. There are currently three delivery types:

  • Online: to be taken by respondents online, via email, website, or otherwise
  • Interviewer: to be facilitated by an interviewer and a tablet computer
  • Display: to be displayed on a device at a set location, like a podium tablet, or a library computer

You need to select all the methods that you will be using to deliver your survey as this will automatically configure some additional settings for the survey and links to different versions, which you will see when you’ve completed the set-up process.

You will be presented with advanced options related to the delivery type(s) you have chosen, including a finish URL (which will redirect respondents to a specific website on completion), a timeout (which will reset the survey if the respondents is inactive for a period of time) and access to survey tools (which will enable interviewers to access tools to support the interview process).

 

Designing your survey

Designing a survey is the process of adding questions and content to your survey. On the Design screen you can add, modify or remove questions.

Adding survey content

In order to add content, simply click and drag the type of question or content into the survey section. We recommend using four to six dimensions which are relevant to your evaluation, but feel free to use whatever mix of questions suits your needs. If you want to learn more about best practice for making surveys in Culture Counts, visit our resources section for more information.

The types of content available in Culture Counts are outlined below:

Dimension

A standardised statement that respondents can agree or disagree with, rated along a slider. Culture Counts dimensions are based on extensive research in each sector in order to identify the key elements that make up the quality of a event, project, or place. These elements have been translated into single statements or ‘dimensions’ to be included in Culture Counts surveys.

One of the main advantages of using these preset dimensions to evaluate the quality of your event is the ability to standardise measurements across the sector. This allows you to measure your performance over time, or against other organisations nationally and internationally.

Dimensions have a special selector for adding them to the survey. When you click to add a dimension, you will be presented with a list of dimension categories relevant to your organisation or survey type (e.g. Quality, Place, Participatory). To add them, simply click the category you wish to add, select the one you want to add, before clicking "Use selected dimension". After the dimension has been added to the page, it will have a dropdown where you can select variants of that dimension that best suit your survey.

Slider

A simple slider input, where responses are recorded along a slider that ranges between three values. By default, these are "Strongly disagree", "Neutral", and "Strongly agree", but can be customised when the question is added.

Date

Provides a date input where respondents can nominate a date from a calendar.

Yes/No

Gives the respondent two options to choose from, which default to "Yes" and "No".

Number

A basic number input, where numbers can be input by either typing, or changed by up and down arrows. Minimum and maximum values can be specified.

Dropdown

Provides a dropdown list, where respondents can answer by choosing one of the options.

Multiple Choice

Provides a selection of answers, of which respondents can answer by selecting one or more of the options.

Checkbox

Provides a selection of checkboxes, where respondents can choose one or more answers from the items specified.

Short Text

A short text input designed for smaller input types, such as postcodes, or up to 12 characters.

Free Text

A simple text input that allows respondents to type an answer or comment.

Message

This shows a message to your respondents, which is not answered. They are typically used to provide additional instruction or context to respondents.

Editing or removing content 

In order to edit content, click on the question text you wish to edit and type your desired text. Some content types, like dimensions, cannot be edited, as they are standardised. To remove a question, click on the "trash" icon in the top right of the question box. The icon with the four squares allows you to change the ordering of questions within your survey. 

Please note that standard demographic questions – age, gender and postcode – are included automatically at the end of all Culture Counts surveys. You do not need to add these questions to your survey yourself.

 

Inviting peer and self assessors

One of the core functions of Culture Counts is being able to invite and manage peers or self assessors to your surveys. That's why we've made it easy, through our invites page. Simply type in the email of the peer or self assessor you wish to invite in the email field, and press enter to add them.

 

Completing and distributing your survey

Once you have been through they Configure, Design and Invite stages, you have created your survey and will be taken to the survey Summary page. Here you will be shown an overview of the survey, with the number of questions you have included, a list of the self and peer assessors you have added and a summary of the survey results so far. You can return to this page at any time to see how your survey is progressing, and which of your nominated respondents have attended or completed their surveys.

Distributing the survey to the public

The Summary page will show you the links or URLs for your survey. There will be a separate URL for each of the different delivery types you selected at configuration stage. To distribute a public survey, simply copy these links and use them in the relevant locations. For example, for a survey to be administered by interviews, enter the Interview link into the browser of your tablet computers. For a survey to be distributed via email, copy the Online link and paste it into the email you are sending to audience members/visitors.

Before distributing the survey, we recommend you click on Preview so that you can make sure that the survey looks just as you intended it to!

Distributing the survey to peers and self respondents

The easiest way to distribute a survey to self and peer respondents is to use Culture Counts to email invitations to your nominated assessors on your behalf. You will see on the survey Summary page that the system automatically generates a unique link to the survey for each of your nominated self and peer assessors. Each self and peer assessor must be sent his or her own unique link, and it’s easy to do this through Culture Counts. On the Summary page, click the Send invitation button next to your first self or peer assessor. A dialogue box will open to allow you to draft your invitation email. The unique survey URL for that particular respondent will automatically be included in the email.

Note that you can communicate with your self and peer assessors outside the Culture Counts system – for example through your own email account – if you prefer. However, for the system to recognise a survey response as being from a self or peer assessor, and therefore for the data to show up correctly in the reporting pages, at the Invite stage you must enter the email addresses of all the self and peer assessors who you are inviting to take part, and make sure that you send each assessor the unique URL that the system generates for them.

If you are asking self and/or peer assessors to complete both a prior and post survey, you need to invite each assessor to complete the prior survey first. When a prior survey is completed, Culture Counts will automatically ask the respondent when they are planning to attend the event. You should only send a self/peer assessor his or her link to the post-event survey once they have completed the prior survey and attended the event. Culture Counts will notify you on the survey Summary page when an invitation to a post-event survey is due.

 

Seeing the results

Once the Culture Counts reporting module is ready, this section of the manual will include information on how to use Culture Counts to view and interpret the data collected through the various surveys included within your evaluation. In the meantime, you can download the data for an individual survey or an entire evaluation in a CSV file using the Download results button.

Have more questions? Submit a request

Comments

Powered by Zendesk