Prototype evaluation plan, process and results

Happy Monday and start of the week – this blogpost will be the second to last so after that you will get to enjoy one more blogpost! This has been very excited and inspirational course and group assignment – we have learnt a lot about the Solita and its projects but also about human-centered project development! In this blogpost we will go through the third phase of implementation which is the MVP concept slash prototype evaluation plan, process and results. We have created an interactive prototype of our innovation through JUSTINMIND application where we also tested our protype. Our prototype is also our MVP (minimum viable product) where we have implemented environment to create a project. After creating the project, this and other created projects can be seen in the page. User can create a project where are asked project information like project name, dates, description, project contact person and workers and project implementation information e.g. techniques and key words. After this project will be saved. 

Testing a prototype is significant and needed phase of the design and manufacturing process because in the best scenario, testing and evaluation can confirm the good work that has been done, is on the right path or it need to be fixed. Evaluation plan’s idea is to design how to do evaluation process and analyze the results but also to identify already existing or potential faults so that we could be able to make improvements. (Ryan 2013) We chose to use user evaluations because we couldn’t schedule the prototype meetings with employees of Solita.  We thought that it wouldn’t be any problems because we could get the prototype go-throughs done with people that are not working with Solita and actually could get even more answers and ideas than just interviewing couple of Solita people. We planned to do first one or two test iteration prototype testing experiments and base on that, do the needed changes in prototype if needed. We plan to use lab testing of user evaluations which means that users can directly interact with and use the prototype (Ramanan 2020). 

We planned to have six to ten experimental testing of users and do the experiment testing face-to-face but also via Teams because of COVID-19 pandemic. Participants will be from different backgrounds and different educations. We planned not to ask any specific questions but just ask the user to follow and use the prototype as he or she would do it – only thing that we asked for users was to tell what they are doing. We have created a task scenario: users need to start a new journey where users create a project for sharing project information to organization’s employees (Ramanan 2020). This could help us to understand users’ thoughts better and make the needed changes or additional features. All the information (qualitative data) that we could get from the users, would be written down and summarized together. 

First iteration of prototype evaluation process: 

Evaluation process started with one test person – with this user feedback and information we could improve the prototype if this test person would find anything specific things to change. We had our first prototype test face to face as lab test where we put the test person front of the computer and opened the interactive prototype and asked test person to create a project following the instructions. We asked user to say what he/she would be thinking and doing and that they to get the insights and after the test we asked overall feedback. Just like in all prototype tests, users needed to create a new project and fill the needed information. After that the project can be saved and viewed just like other projects that have been created. Our first test went well and the user created a project successfully – test user thought that the prototype was easy to use and very clear but had a few suggestions: user suggest to change about the opening the project in different way. We wrote down the feedback but because of where wasn’t any major or essential errors or changes to change, we kept our prototype the same for rest of the test users and second iteration round.

Picture: Example of the project that test user could have been created in test lab environment. 

Second iteration of prototype evaluation process: 

Because of the successful first round of iteration, we continued our second round with the same specs that we had for the first iteration round of evaluation process. We scheduled totally nine test labs with nine different people – one person needed to cancel the timeslot and we couldn’t find a new possible time so from this round we totally got eight user tests. Just like in first prototype test, users needed to create a new project and fill the needed information. After that the project could have been saved and viewed just like other projects that have been created. We have gathered user feedback and change suggestions to the table below. 

Table: User feedback from iteration of prototype evaluation process. 

As we can see from the table, users have been happy with the usability of prototype in general. Users liked that the prototype was easy to use and simplified so where wasn’t too much stuff – users’ opinions are important to understand and taken under consideration so that employees could get the same clear and easy to use experience. It’s not always simple to implement when this prototype should be linked and added to already existing interface. One thing that many users noticed and said out loud was that the starting a project or creating a project would need to be easier to find by maybe implementing images or icons. This is a very significant observation because creating projects are the main tasks in our innovation application and one of the first steps to do. 

Hypotheses: 

We had made our hypotheses beforehand and now after testing the prototype, we can also compare these hypotheses with the results.  Our first hypothesis was “it is easy to find and read the project information in the prototype” and the second “it is easy to add and edit project in the prototype”. Our first hypothesis was confirmed through the evaluation process because all the users really though that it was easy to use in general and information were also easily presented. The second hypothesis was partly confirmed – users felt that creating and adding the project happened very easily but at least one user said that the editing afterwards wasn’t easy so this could one possible thing to focus on when designing this prototype in the future - employees have plenty of work to do and if we want to help them to success on their work, work time should not be wasted to find fundamental and basic stuff. This requires a lot of knowledge of system contexts and organizational behaviour so that the the prototype could be implemented in a right direction (Edwards et al. 2005). 

Before this protype can be taken more further, it should be kept testing and evolving. It should be tested with the employees but also in the HR system context where our prototype would probably could belong. Also this prototype should be tested against needed regulations and legislations (Ryan 2013). Overall, our group found this prototype phase very interesting and instructive! Interactive prototype really showed how our innovation from the sketches was really developed to this “real” and concreate interface. This was definitely one of the highlights of this whole course and concreate our hard work during the long process. 

(Green) peas and love, 

<3 KBD 

Link to our interactive prototype: https://cloud.justinmind.com/usernote/prototypes/51775599/51776065/51776067/index.html#/screens/f0fd1826-73ec-48d8-a41c-db45ee161bf8 

References: 

Edwards, M., Peer, R. & Lindner, E. (2005). Evaluating Cultural Resource Significance: Implementation Tools. Chapter 3. NCHRP Report 542. National Cooperative Highway Research Program. Available at: https://www.nap.edu/read/13815/chapter/1 

Ramanan, T. (2020). Prototype Evaluation Plan – for the design of: Commute Communities. Available at: https://static1.squarespace.com/static/56720ffb05f8e24f35063361/t/56a86a3ac21b869a7c0f1968/1453877820563/psyc-hw3_neutral.pdf (read 24.4.2021) 

Ryan, V. (2013). Testing and Evaluating a Developed Design / Prototype. World Association of Technology Teachers. Available at: https://technologystudent.com/despro_flsh/evalintegr1.html (read 24.4.2021)