Evaluating programming systems design

The Journal of Computational Science Education, 2015

In this paper we present an iterative research process to integrate worked examples for introductory programming learning activities. Learning how to program involves many cognitive processes that may result in a high cognitive load. The use of worked examples has been described as a relevant approach to reduce student cognitive load in complex tasks. Learning materials were designed based on instructional principles of worked examples and were used for a freshman programming course. Moreover, the learning materials were refined after each iteration based on student feedback. The results showed that novice students benefited more than experienced students when exposed to the worked examples. In addition, encouraging students to carry out an elaborated self-explanation of their coded solutions may be a relevant learning strategy when implementing worked examples pedagogy.

2021

Learning a programming language is a difficult matter with numerous obstacles for university students – but also for their lecturers. Assessment tools for programming assignments can support both groups in this process. They shall be adapted to the needs of beginners and inexperienced students, but also be helpful in long-term use. We utilised an adapted version of the Artemis system as an assessment platform for first-year computer science students in the introductory programming course. To examine the students’ user experience (UX) over the semester, we conducted a three-stage online questionnaire study (N=42). We found that UX evolves over the semester and that platform requirements and problems in its usage change over time. Our results show that newcomers need to be addressed with caution in the first weeks of the semester to overcome hurdles. Challenges shall be added as the semester progresses. 2012 ACM Subject Classification Human-centered computing

CHI '07 extended abstracts on Human factors in computing systems - CHI '07, 2007

From 18.01.07 to 23.02.07, the Dagstuhl Seminar 07081 End-User Software Engineering was held in the International Conference and Research Center (IBFI), Schloss Dagstuhl. During the seminar, several participants presented their current research, and ongoing work and open problems were discussed. Abstracts of the presentations given during the seminar as well as abstracts of seminar results and ideas are put together in this paper. The rst section describes the seminar topics and goals in general. Links to extended abstracts or full papers are provided, if available. (IBFI), Schloss Dagstuhl. During the seminar, several participants presented their current research, and ongoing work and open problems were discussed. This document summarizes the event. Although gender dierences in a technological world are receiving signicant research attention, much of the research and practice has aimed at how society and education can impact the successes and retention of female computer science professionals. The possibility of gender issues within software, however, has received almost no attention. We hypothesize that factors within software have a strong impact on how well female problem solvers can make use of the software. Evidence from other elds and investigations of our own have revealed evidence supporting this hypothesis. Towards the end of the 20th century, a paradigm shift took place in many scientic labs. Scientists embarked on a new form of scientic inquiry seeking to understand the behavior of complex adaptive systems that increasingly deed traditional reductive analysis. By combining experimental methodology with computer-based simulation tools, scientists gain greater understanding of the behavior of systems such as forest ecologies, global economies, climate modeling, and beach erosion. This improved understanding is already being used to inuence policy in critical areas that will aect our nation's future, and the world's. How does EUSE research build on empirical studies of programmers, and what kinds of empirical research might provide foundations for future EUSE research? End-User Software Engineering 3 My own work on interdisciplinary design draws comparisons across academic and professional boundaries, applying the results to the design of new technologies, and the critical assessment of technology.

2002

A programming system is the user interface between the programmer and the computer. Programming is a notoriously difficult activity, and some of this difficulty can be attributed to the user interface as opposed to other factors. Historically, the designs of programming languages and tools have not emphasized usability. This thesis describes a new process for designing programming systems where HCI knowledge, principles and methods play an important role in all design decisions. The process began with an exhaustive review of three decades of research and observations about the difficulties encountered by beginner programmers. This material was catalogued and organized for this project as well as for the benefit of other future language designers. Where questions remained unanswered, new studies were designed and conducted, to examine how beginners naturally think about and express problem solutions. These studies revealed ways that current popular programming languages fail to suppo...

User-centered design (UCD) guides user evaluation and the design of the user interfaces (UI) as part of the software development process. In this paper, we present our work to integrate UCD activities into software development environments. Specifically, we present an Eclipse plug-in that is under development that its main capability is to manage the UCD activities within the development environment thus providing Eclipse perspectives for the evaluation manger role and for the UI designer role enabling defining and executing UCD experiments, viewing experiments results, and manipulating the implied UI design tasks. We manage six teams of six students in each team who work to develop the plug-in. We report on the first iteration of development including the requirements and parts of the high level design. This work emerges from the DELOS tasks that relate to user evaluation techniques.

2003

This paper describes an analysis scheme which was developed to probe the comprehension of computer programming languages by students learning to program. The scheme operates on free-form program summaries, i.e. textual descriptions of a program which are produced in response to minimal instructions by the researcher/experimenter. The scheme has been applied to descriptions of programs written in various languages, and it is felt that the scheme has the potential to be applied to languages of markedly different types (e.g. procedural, objectoriented, event-driven). The paper first discusses the basis for the scheme, before describing the scheme in detail. It then presents examples of the scheme’s application, and concludes with a discussion of some open issues.

1996

: This report reviews and organizes research about novice programmers. Over the past two decades, many aspects of novice programming have been investigated, resulting in the discovery of important facts and tradeoffs about what makes programming difficult to learn, and about the effectiveness of existing languages, environments, and methods of instruction. However, because this research is dispersed throughout the literature, it is difficult for designers of new programming systems to consider all of the issues collectively. The result is that most new systems are built primarily around technical objectives, perhaps considering only a subset of the usability issues summarized here. In addition to providing a checklist of issues that should be considered in the design of future systems, this report can be used to help researchers identify fruitful topics of future novice programming research.

Proceedings of the 14th International Conference on Computer Supported Education, 2022

In this paper, we analyse the educational design research process in which we looked to develop an evidence-based intervention and professional development (PD) content for programming in primary Year 4b. We ex-plain how the latest period of the Computing with Emil project extends our previous intervention for Year 3 by exploiting the same constructivist pedagogy and design principles. We recall these principles and the re-search strategy which we consider appropriate for the current state of knowledge in transforming computing education in lower primary years. As the concluding phase of the design research, we reflect on the design process in greater depth and formulate the characteristics of the intervention by focusing on its programming language and concepts. To frame this effort, we identify five interwoven contexts that shaped the development cycles of our inquiry, namely: (1) programming environment, (2) programming language, (3) content devel-opment, (4) workbook for pupils, and (5) teachers’ PD design. Looking back, we analyse the iterative design process within these contexts through the lens of the five instruments of reflection we were continuously exploiting to inform our design, namely: (a) observing pupils work, (b) interviewing teachers, (c) analysing pupils’ workbooks and onscreen content, (d) evaluating Emil Y4 pilot PD sessions, and (e) analysing Emil Y3 data. Among other findings, which have resulted from the analysis of the design process, we highlight in particular a thoughtfully designed pulsating range of the command set. Although we had gradually created the content ourselves, it was not earlier than the concluding retrospective analysis that exposed it so clearly.

Interacting with Computers, 2001

End user programming (EUP) environments are difficult to evaluate empirically. Most users do not engage in programming, and those who do are often discouraged by the complexity of programming tasks. Often the difficulties arise from the programming languages in which users are expected to express themselves. But there are other difficulties associated with designing extensions and adjustments to artifacts that have been originally designed by others. This paper characterizes EUP as a semiotic design process, and presents two principles that can be used to illustrate the distinctions between the various kinds of techniques and approaches proposed in this field. The principles support a preliminary theoretical model of EUP and should thus facilitate the definition and interpretation of empirical evaluation studies. They also define some specific semiotic qualifications that more usable and applicable EUP languages could be expected to have. ᭧ Souza). its product and continued evolution. We will argue that this is a preliminary requirement for significant quantitative and qualitative usability measurements in this context, even more fundamental than user-centered methods that could be used to evaluate tailorable applications (see Malone et al., 1995 for instance).

Human Technology An Interdisciplinary Journal On Humans In ICT Environments

The catch phrase today for system designers is to "make it intuitive," which begs the question, what is intuitive? The action research discussed in this article was the final stage of the application of grounded theory to user data that provided survey categories (criteria) for system acceptance. A theoretical rationale from the discipline of human-computer interaction was proposed to provide a consistent and repeatable interpretation of the users' responses to the survey and directly align the responses to software design considerations. To put this work into context, I discuss in this article a case study on the use of the survey to monitor the user experience during the upgrade of an enterprise system and the subsequent implications and outcomes of applying the theoretical paradigm in practice. As such it may provide food for thought on survey design for elicitation of user requirements for information and communication technology systems.