Technology Evaluation using Empirical Methods

Engineering-style software development requires technology whose suitability has been systematically evaluated and whose costs, risks, and benefits are known. Empirical studies are an approved means for systematically evaluating benefits and drawbacks in a practice-oriented manner. However, professional studies require systematic, goal-oriented planning, execution, and analysis.

New technology must be tested before being introduced into practice. Insufficiently optimized, inefficient, or faulty technology may severely disrupt active development processes and demotivate employees.

Evidence-based technology evaluation checks usability and suitability beyond mere technical aspects and indicates potential areas of optimization.

This seminar serves to acquire a basic understanding of how to systematically evaluate technologies using empirical methods. It teaches the core empirical process to be followed and how it is used in practice. This includes how to plan, design, implement, and execute studies, analyze and report findings, and appraise the quality of a study.

Introduction and Motivation

• Technology Evaluation
• Overview of Empirical Methods
• Application Scenarios
• Benefits
• Limitations

The Empirical Process

• Definition
• Design
• Implementation
• Execution
• Analysis
• Packaging

Exercise: Participating in a Technology Evaluation Study

Classes of Empirical Methods and Examples

• Surveys and Interviews
• Case Studies
• Experimental

Toolbox for Empirical Methods

• Basic Statistics
• Hypotheses Testing
• Software Tool Support

Exercise: Designing an Empirical Study

Summary and Conclusions

• Benefits versus Cost
• Lessons Learned