Evaluation Resources

Program working Team:
● Don Chan – Operations Lead for learning activities across the business
● Dave Wilde – Operations Lead for a sub-business; in charge of program/effectiveness
● Christine Vance – Learning & Development consultant
● Janet Greathouse – Learning Solutions Manager (SME for Agile)
● Peter Vollmer – Employee, Instructor SME of current courses
● Dan Paige – Business Leader

Leadership has been engaged at the SVP, VP and Director levels within the business in order to align business needs to the learning objectives of the program. Leaders have a confirmed committment to participate in a first draft leadership training prior to their teams in order to communicate, understand and implement the changes to the business according to the new style of project management.
Additional Stakeholders
Name Program Involement
1,500 Employees
within Automate & Optimization business group Intended audience; will provide feedback to program
Operations Leads Liason to re-confirm prioritization list of 1,500 employees for Q4 – Q2 training; provide input for what is important to evaluate.
Leadership Staff Financial sign off, buy-in at SVP level/sponsorship, understand evaluation results to make future decisions, etc.
Chief of Staff Financial – budget responsibilities to business (eg involved in approval for course deliveries, travel for course, any cost for evaluations)
Instructors Delivering courses; can provide some feedback and also weigh in on any future SME instructors from courses
Managers of Attendees Provide sponsorship to attend course and utilize learnings in team; help nomination employees and potentially future certified instructors from current employees/managers
Potential adopters of the program (broader HP Enterprise) Will make operational decisions at higher level than Software business from evaluation business impact results

Purpose
Once stakeholders are engaged and the evaluation team has gathered the opinions of all parties involved, it is necessary to define the reason for the evaluation. The evaluation team may decide, based on input from stakeholders and available budget in place, to look at the adequacy of supply chain distribution. The purpose of this evaluation will be defined in conjunction with the evaluation team and input from key stakeholders. Defining the purpose allows evaluators to create a substantial, measurable evaluation question upon which the entire evaluation process is centered upon. The evaluation shall answer the following questions (Kirkpatrick & Kirkpatrick, 2006, p. 75):
● Who are the key stakeholders?
○ In charge of the program?
○ Development of the program?
○ Who Those program is intended for?
● Is there anyone we should involve to understand KPIs or what success looks like?
● Who / what resources are available?
● What is the business need and how is this program going to meet that need?
● How was the solution identified?
● Is there leadership sponsorship to help ensure rollout success?
● What would success look like?
○ What are short term goals, intermediate and long term goals? (knowledge, skill, attitude)
What are the standards to judge the quality of the program?
● What decisions will the program managers need to make based off of our eval results?
● What is the goal of having an evaluation plan?
Approach
Based on interviewing the company’s program managers, to gain the most value add from collecting data, a participant-oriented approach should be taken for the evaluation plan. In this approach participants (stakeholders) are central (key) in determining the values, criteria, needs, data and conclusion for the evaluation.
Type of methods to used within this propsal are a combination of participant self-assessment, stakeholder evaluation, internal evaluation and joint evaluation – brainstorming on program strengths and weaknesses.
Reasons to use this approach:
● Involves stakeholders in the process of an evaluation can lead to more valid data, understanding of the data, and more appropriate recommendations.
● Provides clear guidelines of what purpose will stakeholder participation serve in this impact evaluation?
In addition there are some challenges to using the participant-orientated approaches such as:
● Time and commitment- collaborations of business schedules for the purpose of brainstorming.
Level 1: Reaction
The first stage of the evaluation program will involve a survey provided to employees during the face to face sessions prior to their departure to ensure 100% participation to analyze most holistic, valid feedback (Kirkpatrick & Kirpatrick, 2004). This level of assessment will help determine: the reaction of the participant to the course, instructor, logisitics, learning environment and relevance of information. By answering questions associated to each of these areas, program managers and leaders will be able to adjust at a high-level learner experience the course delivery and content. The below evaluation is already implimented throughout HP Enterprise as standard through Metrics That Matter using a 5 pt Likert rating system.
Post Course Learner Evaluation
Relevance to the Job
The course was relevant to my job/role.
This training was a good use of my time.
Learner Satisfaction
This course met my expectations.
Please rate your overall satisfaction with this course.
Instructor
The instructor’s teaching methods, style, interpersonal skills and pace helped me to learn.
The instructor’s subject matter expertise enhanced my learning experience.
Logistics and Classroom Environment
This delivery method was an effective way for me to learn the material.
The classroom environment (room, lab, equipment, network, etc.) functioned properly.
Net promoter/Satisfaction
How likely are you to recommend this training to a friend or colleague?
How does this course compare to similar courses you’ve taken?
Please rate the overall quality of this course.
Learner Effectiveness/Job Impact/Business Results
This training will help me:
● improve quality of my work
● improve speed of my work
● improve how I interact with others
● improve my leadership skills
● improve how I meet my customer’s needs
● improve how I meet my performance goals
How much of this training do you plan to use on your job? (rating 0% – 100%)

Level 2: Learning
Questions asked
The project can go on well when the team has questions that guide in the brainstorming of why there needs to be a change in the way Software project manages their product development, testing and go to market deadlines across multi-channel teams. The number that is conversant with the acronyms in software should be known so that members who are below par catch up with the rest. Questions about a project will act as a point of remembering and evaluating what knowledge the team needs vs. what they already have. These are some of the sample questions we propose to use:
Q1. What is the current state of knowledge of participants?
Q2. What information do the participants need to gain?
Q3. What is the best method that can be used to make the team members learn about project management methodology change?
Methods of data collection
After conducting an evaluation assessment, there is a need to have the results so that the program team prepares well to deliver what is lacking training for identified knowledge gaps. The choice of the data collection method should be aligned to the objectives and performance indicators of the team. The test procedures include:
1. Proficiency test- it measures the skills and knowledge of the participants before and after attending any training course
2. Questionnaires- the questionnaires can be closed or open ended. The closed questions will do better because the team has a common objective. The choice can be Strongly Like, Like, Dislike and Strongly Dislike, aligning to a Likert scale quantitative model.
Besides the methods, data can be collected by interviewing the participants and making observations. The observations can be made on their body language, facial expression and the gestures (Magee, 2006). Group assessment may not prove handy because the interviewees can give general opinions given by the other members. The results are recorded, and the scores indicated on a score sheet. Recording the subsequent scores will get the learning change and hence the percentage. The following formula is used:

Source of formula (Napier et al, 2012)
Reason for learning gain
There is learning gain because most of the participants have had from the team and they have been equipped with the information that was lacking. The change is due to the learning that takes place inside the classroom. Care should be taken that the participants do not give leading answers so that they record a positive change even when they have not gained from the forum.
Note: We are in the process of asking client the following questions which will help determine next steps for Level 3 and 4.
– What are short term goals, intermediate and long term goals? (knowledge, skill, attitude)
– Increase skill and knowledge of agile methodology for key personnel
– Change attitude of leadership to come onboard for this pivot to new model
– What decisions will the program managers need to make based off of our eval results?
– PMs will need to use results to see productivity level changes (e.g. decreased speed to release products and increase net promoter scores) after someone has taken these classes vs. those who have not. They need to understand the business impact to see if this program should be offered to all technologists working at HP Enterprise.
– They also will need to use data to adjust program layout, courses offered, instructors, etc. for best quality to learners
– What is the goal of having an evaluation plan? What information do stakeholders need or hope to gain from the evaluation?
– From the evaluation, people are expressing the need to see the behavior change in the key technologists who are working on product rollouts due in December. These folks are used to having 9 months to develop, test and rollout a new software product, and now the business model has shifted to every 3 months to be more agile for the needs of the customer (e.g. we needed it yesterday).
– What requirements does the evaluation need to fulfill?
– Test productivity levels
– test knowledge before and after gained/not gained
– have control group to measure against of those who did not take the classes
– have baseline productivity levels to measure against
– Which components of the program are the strongest candidates for evaluation?
– The two core classes being offered are the bulk of the program. It may be hardest to measure attitude change of management.
– How will the program benefit the company?
– Meet the demand of the customer and market competitors to teach technologists (e.g. engineers, product manages, etc.) how to increase productivity levels and decrease time to market.

Level 3: Behavior
Behavior analysis is conducted to confirm which of the skills taught to the employees are being applied. The analysis is performed to determine whether the training is practical and appropriately applied (Kirkpatrick & Kirkpatrick, 2006, p. 42).
1. Implement control group with strongest design.
2. Allow time for the behavior change to take place.
3. Evaluate both before and after the program.
4. Survey and/or interview one or more of the following: trainees, their immediate supervisor, their subordinates, and others who often observe their behavior.
5. Ensure that 100 percent response and sampling is obtained.
6. Repeat the evaluation at appropriate times if necessary.
7. Evaluate costs versus benefits (Kirkpatrick & Kirkpatrick, 2006, p. 64).

Level 4: Results
This is the final results phase of the evaluation proposal. This phase monitors and measures the training program’s effectiveness, that is, “What impact has the training achieved?” These impacts can include such items as monetary, efficiency, productivity levels, teamwork, etc.
As we move from level one to level four, the evaluation process becomes more difficult and time-consuming, however, the higher levels provide information that is of increasingly significant value (Kirkpatrick & Kirkpatrick, 2006, p. 72).
• Did the learner apply the new skills to the necessary tasks in the organization and, if so, what results were achieved?
• Measure organizational results before and after training the training group
• Measure results of control group who did not participate in program against those who did
• Compute the change in organizational results
• Organizational results could include anything that the training might affect including profits, costs, productivity, quality, employee morale, etc.
• Overall, Did the employee’s job performance improve?
1. Implement a control group.
2. Allow time for results to be achieved.
3. Measure both before and after the program if practical. Implement the strongest experimental design.
4. Repeat the measurement at appropriate times. Repeated measurement (e.g., using the time-series design) can provide data on the long term pattern of results.
5. Consider costs versus benefits (Kirkpatrick & Kirkpatrick, 2006, p. 66).
Evaluation Schedule
Evaluation Deliverables Effort
(Hours or Days) Resources
(Who & What) Delivery Methods
(How to Deliver the Deliverables) Delivery Date
Initial team meetings – decide company to ask 1 hr All team members decide on work split Conference call Week1
Initial company commitment 2 hrs Christine – Hewlett Packard Enterprise Face to Face Meetings Week 1
Team Check In for Client questions & Draft 1 hr All team members discuss answers and sections completed Conference call Week 2
Questions answered from client 2 days All team members:
Christine to deliver Email Week 3
Draft version 2 1-3 days All team members turn in and agree to sections Call / Email / Google Docs Week 5
Edits 1-3 days All team members edit draft document; incorporate feedback Call / Email / Google Docs Week 6
Final Version 1-3 days All team members finalize together proposal; incorporate feedback Call / Email / Google Docs Week 6

Costs
According to Fitzpatrick, Sanders, and Worthen (2004), we have noted some of the costs that might incur for our evaluation plan. Per Christine our cost should be low based on internal people working / participating in the program, but we will need to decide what costs if any to the items below will need to be budgeted and accounted for to the client to ensure we align to expectations. In accordance to Level 4 of the eavluation, we will measure ROI the completion of the program to show the business impact of such an investment and how understanding all feedback is important for improvement , efficiency and meeting business needs when having this type of program in the future. Summarized below are the areas where cost might be addressed.
Evaluation Costs
1. Evaluation staff salary & benefits
a. Cost in this category are relatively easily by normal using existing salaries
2. Consultants
a. If need to provide skill not currently reflected in the project program
3. Travel and per diem
a. If there is any field work on this project/program then item #3 will need to be addressed.
4. Communications
a. Postage, telephone calls etc. for example mailed surveys etc.
5. Printing
a. Cost covered for preparation of any printing item related to the project/program would do here.
6. Data Processing
a. Will we purchase software for data analysis, storage retrieval
7. Supplies and equipment
a. Any equipment rented or brought for the purpose of the program/project even pens, pencils, paper etc.
8. Overhead
References

Jones, S., In Graham, S., & IGI Global,. (2015, Feb 28). Cases on sustainable human
resources management in the Middle East and Asia. IGI Global, 1st ed.

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2010). Evaluating training programs: The four levels.
Sydney, NSW, Australia: ReadHowYouWant.com.

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating reaction. In Evaluating training programs: the four levels. Berrett-Koehler.

Magee, K. (2006). Body language. Denton, Tex: University of North Texas Press.

Napier, H. A., Rivers, O. N., & Hoggatt, J. (2012). Learning with computers: I. Mason, Ohio:
South-Western.