Runestone Academy gets Math
Two important announcements this week!
Math books on Runestone academy!
Runestone Academy will host WebWork courses
Read on for the details…
It all began last December at the American Institute for Mathematics Workshop on OER assessments. Its a partnership that makes a lot of sense as our goals and values are so clearly aligned around making great textbooks available to students for free. Whereas the PreTeXt project has put a lot of focus on the write once and publish in many different mediums paradigm, Runestone has put more effort into developing an html only interactive textbook with an integrated LMS offered as a service. PreTeXt books can be published as HTML, EPUB, PDF and other formats including braille! Now PreTeXt books can also be published for Runestone.
I’m pleased to announce that we will be hosting several textbooks written using PreTeXt. beginning today you can use the following books:
Open Resources for Community College Algebra (ORCCA) – Faculty of Portland Community College
A First Course in Linear Algebra – Rob Beezer
Abstract Algebra: Theory and Applications – Thomas Judson
Discrete Mathematics: an Open Introduction – Oscar Levin
APEX Calculus (coming soon) – Gregory Hartman, Ph.D., Sean Fitzpatrick, Ph.D. (Editor), Alex Jordan, Ph.D. (Editor), Carly Vollet, M.S. (Editor)
Over the last 9 months I’ve been working with Rob Beezer and the team to integrate PreTeXt books into Runestone. We have things like activecode, short answer, and videos working seamlessly with Runestone. This means that Math teachers can use one of these books on Runestone and get all the benefits of grading, and following student progress. One of our key challenges was getting Runestone to work with WebWork problems. WebWork is widely used in the math world and has an enormous catalog of homework problems.
In order to make the integration work well, we decided that Runestone should host its own WebWork server. This allows us the flexibility to run a custom server that supports the needs of Runestone on our schedule. The load on the server is fairly light for Runestone’s purposes so Alex proposed that we use that spare capacity to serve the community. Alex Jordan is a WeBWorK developer partnering with Runestone Academy for this service, and will be managing the WebWork server and providing faculty with some level of support. Alex has been invaluable in getting the integration complete.
Runestone Academy is hosting free WeBWorK courses for Fall 2020. If you would like to use this service (or if you have questions), Check out our Webwork Page
Active Prelude to Calculus
Active Calculus
Active Calculus Multivariable
ORCCA (basic and intermediate algebra)
This is a great addition to Runestone. We can learn a lot from the WebWork community as they have done an excellent job of evolving their question authoring system over the years.
Why are we doing this? First, we recognize that in the era of remote teaching during a pandemic, some faculty are still searching for an affordable online homework platform. Second, if this semester’s WeBWorK trial is successful, we plan to offer this as an inexpensive but fee-based add-on service to Runestone Academy. We would charge between $50 and $100 per course (not per student). This is part of a multifaceted plan to make Runestone Academy sustainable. All revenue exceeding expenses would support some open source educational technology, such as Runestone and the WeBWorK Project.
In case you did not know, WeBWorK problems and problem sets are easily portable. Any PreTeXt book with WebWork can export and archive of problems for use ona WebWork server. If you try this service in the Fall and would like to move to some other server once Runestone academy starts charging for the service, you will be able to export everything with no hassle.
If you are new to Runestone, this YouTube Playlist will demonstrate every aspect of Runestone from the instructors side. To get a good idea of the student experience you can take a look at The Overview Book The PreTeXt project also has a number of samples available.
New Titles
This week I’m really proud to announce that I’ve been promoted to “Senior Executive Bartender.” Ha, If you got that reference congratulations you are old enough to have been a fan of Cheers! If you don’t know what Cheers is, then we will just move along.
But seriously folks I really am excited to announce two new books available for your courses this Fall. The first is How to Think like a Computer Scientist C++ interactive edition. This is an Allen Downey book in the same basic pattern as the thinkcspy book on Runestone but in C++. Barb Ericson and her students at the University of Michigan (The other U of M) have been working on doing the conversion to an interactive eBook this summer and it looks great, and really exercises all of the work we have done on improving C++ activecode, CodeLens, and unit tests for C++.
A few people have asked me about the Java version in this series, and all I can say is that I hope someone will take up the task! I think that with the work Barb and her students have done here, and the fact that the books are quite similar that it would be easy to adapt the C++ to Java. Any takers?
The second new book Mastering Relational Databases by Libby Shoop at Macalester College has a wonderful back story that fills my heart with joy. This is a great new book that I hope many undergraduate institutions will adopt. It is also the first major work to feature SQL in the browser with unit tests! But thats not the reason I am so happy to see this published? I’ll let Libby give you a preview:
A great deal of the material in part 2 was originally written by John Carlis for an audience that included a wide range of people, from undergraduate and graduate students to experienced computing professionals. His unpublished manuscript was entitled “Mastering Database Querying and Analysis”. Its primary focus was relational algebra precedence charts. In 2015, three years before his death, he had given me permission to edit a copy of his material to make it approachable for undergraduate students. This online book is the result of my attempt at that task, most of which I am completing after his death.
John was my Masters advisor at the University of Minnesota (the *real* U of M), and an amazing mentor to many many students. He was Libby’s PhD adisor, and we crossed over for a few years when we were both John’s students. John was the person who encouraged me to write my first book. He was an amazing teacher, mentor and huge proponent of active learning and practicing. I believe he would be very happy to see this book come to life on the Runestone Platform! I’ve written more about John’s influence on my life’s path on my personal blog
There are more new titles in the works, but I’ll save them for another day. Please help us spread the word and tell your friends and colleagues.
Runestone Funding
In 2018 I took advantage of an “early retirement” opportunity from Luther College to pursue working on Runestone full time. That opportunity afforded me a year to figure out a business model for Runestone so that I could continue to give Runestone my full attention. However, before the end of the 2017/2018 school year, I got an even cooler opportunity to work with a group Google working to bring computer science education to underserved institutions. Google would use Runestone as the platform for the Applied Computing series and I would work with them, developing both content and features for Runestone. This contract was originally written for five months, but ultimately extended to the maximum contract length of two years. Earlier this month the contract came to an end.
During this time period Runestone has experienced tremendous growth, pushed further this last spring as everyone had to move online. Since the fall of 2018 over 116,000 students have registered on Runestone.academy! That is a lot of students, and if you do the math, we have saved students around the world millions of dollars in textbook costs. Just last week we hosted an online workshop for Runestone instructors with over 240 people registered! If you missed it you can find the videos on our YouTube channel
While the last two years have been exciting, fun, and profitable, consulting for Google really isn’t a business model for Runestone. So, for the last many weeks I’ve been having conversations with some of the major stakeholders and contributors to Runestone including authors, and code hackers, and instructors like most of you. We have big hopes and dreams to grow the platform! Today I write to tell you the outcome of these discussions and how that will effect Runestone moving forward.
My goals for a funding model for Runestone include:
Keep Runestone books FREE for all students. This has always been the North Star for me.
Provide a first class educational experience for all students using Runestone.
Make Runestone sustainable. In other words, cover the server costs, pay myself a small salary, ideally provide for summer internship opportunities for a few students.
Provide a solid foundation to continue the growth and development of the Runestone platform. More authors, more books, better features. For example
Developing more user friendly tools to encourage new authors.
Creating additional opportunities to customize textbooks.
Funding for workshops for new instructors, hackers, and authors
Continuing to develop innovative new features for Runestone books
Develop communities of practice around each textbook where all instructors and authors can contribute ideas, teaching tips, assignments, teaching materials.
We considered many different models, and wow is it difficult to find a model that works for everyone given the wide range of people that use Runestone. We serve everyone from independent learners to public high schools to large universities. But starting soon, here are the three mechanisms we are going to try.
All pages viewed by non-registered users will contain an advertisement. Around 40-50% of the traffic on Runestone is realized as the result of a Google search. These users typically view one or two pages and then move on. But the volume of this alone is quite large. Let me be super clear When your students login and are viewing your course, they WILL NOT see an ad! This is to generate money from the large number of transient users of the Runestone pages. If your student sees an ad that should be a very clear indication that they are not logged in.
Beginning this Fall, we will do a WikiMedia style fundraiser a few weeks into the term. By this I mean we will display banners to registered students that have not chosen to support Runestone. If your class is identified as a public high school course they will be automatically excluded from this as I realize it is not appropriate. Less than 1% of students choose to support Runestone at registration time. However they haven’t experienced a Runestone book yet. Given a few weeks and the value that we provide I believe we can increase that percentage significantly by asking students a few weeks after they have used the books. This ask will be limited in time so we will not nag them forever. Just a week or so.
I will continue to work with researchers at academic institutions to participate in grants. Getting grant money to support the development of significant Runestone features is an awesome way to fund that development. I would welcome the opportunity to partner with any of you that are writing grants and using Runestone in your research or classes.
I think these are some of the least intrusive ways we can generate diverse revenue streams that will help us realize our goals. First and foremost I want to ensure that Runestone is robust and available for years to come. I welcome your feedback, thoughts and further ideas.
Thanks,
Brad
Runestone Workshop June 22-25 2020
Barb Ericson and I are pleased to announce that our scheduled SIGCSE Workshop will now take place virtually, and you are all invited to participate free of charge. The workshop will take place the week of June 22, Monday through Thursday and will use a blended format of asynchronous videos and assignments along synchronous discussion sessions. The live discussion sessions will happen at 3PM EDT. You are welcome to attend any or all of the sessions depending on your schedule and previous experience with Runestone.
To Sign up for the workshop please visit https://runestone.academy If don’t have an account please make one and register for the course UsingRunestone20. If you already have an account on Runestone then add yourself to UsingRunestone20. Note this will enroll you in the Foundations of Python Programming course which we will use as a starting point. We will also spend a lot of time with the instructors guide.
Module One - Runestone Student Experience
Introduction
goals for the workshop
history of Runestone
Be a student
enroll in course
check out your first assignment - reading assignment
checkout your second assignment - problem set
Using the Practice Tool
Module Two - Teacher Experience Creating a course
Creating a course (base course versus bespoke course)
Student Management
Enrolling and dropping students
Adding a TA
Monitoring Student Progress
Instructor only features of the textbooks
Setting up practice
Module Three - Creating your own Assignments Writing and Customizing
Creating assignments and questions
Making assignments
Reading Assignments
Problem Sets
Grading assignments
Using the autograder
Manually grading
Editor Role and flagging bad questions
Module Four - Stories from the Instructor Community
Discussion with guest instructors
Creating a community of practice
Runestone 5.0
Today I am extremely pleased to announce Runestone 5.0! This is a combined new release of both the server as well as the components. Not a lot of this release is highly visible but there is a lot going on behind the scenes.
Perhaps the biggest change for everyone is that pages should load faster. We have moved to using the webpack system to bundle all of our javascript files into a single runestone.js file. Pages used to grab more than 100 javascript and css files, but that has been reduced to just a few now. This makes Runestone MUCH more interoperable with things like Markdown or vanilla html. It was also a really good chance to clean up, refactor and modularize a lot of ugly javascript that had accumulated in the last 9 years!
For Instructors
The Runestone instructor interface has accumulated a lot of great changes in the last 9 months. I attribute many of these changes to wonderful discussions I have had with teachers on the CSAwesome mailing list.
I have set up Google Groups for each book, with the goal of using these groups to develop communities of practice around each book. There is a lot we can all learn from each other and share with each other. These groups will also serve as the gatekeeper for access to instructor-only materials. In particular both FOPP and HTTLADS have very highly developed instructor manuals now.
Summer is a great time to get involved
We have a lot of projects in the pipeline for this summer! Including lots of great new content for you to use this Fall. Here are some links to help you get started.
Installing the Runestone Components and building starter book
Installing the Runestone Server with Docker
A quick intro to a Docker/Runestone Workflow
We can use help writing unit tests for exercises, editing exercises, adding new features, whatever your interest there is sure to be a project for the common good that you can get involved with. You don’t need to be an expert, none of us were when we started, and we are happy to help get you up and running.
Runestone Partners with Edfinity to bring homework system to CS1
Today I’m pleased to announce Runestone’s partnership with Edfinity, our first commercial partnership! Edfinity provides affordable access to exceptional homework problems and assessments customized for any textbook. While Edfinity has focused on the Math and Physics markets thus far they have chosen to partner with Runestone as they move into the computer science world. Edfinity was created with the NSF support (Awards #1758301 and #1943530) - they are best characterized as a for-profit social venture with equal focus on educational research and educational technology. Since the launch of their Math/Physics homework in Fall 2019, over 125 institutions have adopted Edfinity for one or more introductory courses.
Shivram Venkatasubramaniam, CEO of Edfinity has this to say: “Runestone offers high-quality instructional content on a mature, open platform for introductory computer science that has gained adoption at hundreds of institutions across the US. They have a relentless focus on educators needs with a view to delivering superior learning outcomes, and a vision borne out of their deep experience as educators. We, therefore, view Runestone as a natural complement to the Edfinity homework/assessment platform - together Runestone and Edfinity are very well-positioned to serve the needs of educators and students at scale - not just for Computer Science, but other STEM subjects as well. Runestone’s value proposition as an open, interactive textbook platform applies equally to other STEM disciplines”
In eight years of providing textbooks to the computer science community the Runestone team has learned that even for a free textbook, that is only the beginning of what schools are looking for. Many instructors are looking for a commercial class homework system to compliment the Runestone textbooks. The partnership with Edfinity will fill that void. Instructors, especially those teaching CS for the first time, need pre-packaged assignments and assessments. Over the next few months we will be working with the Edfinity team and the community to develop homework and assessment packs that not only complement the How to Think like a Computer Scientist and Foundations of Python Programming text books, but can be easily paired with any commercial textbook for a fraction of the price. A math educator commented recently - ‘Edfinity is as close to free as you can get for a commercial quality homework system’ - I expect this to be true of the Edfinity/Runestone CS homework offering as well. Longer term we will work together to make the experience of using Runestone with Edfinity as seamless as possible.
As an immediate first step, Edfinity is funding several paid positions for undergrads, faculty and high school students (openings and postdoc - please contact me if you are interested) to help write and test exercises, unit tests, quizzes and exams to accompany the textbooks. I’m really excited for this opportunity to work with the team at Edfinity, it’s clear that they are laser focused on bringing a world class homework and assessment system at an affordable price to instructors that really need it. To see how Edfinity works you can check out some of their math offerings on their website at edfinity.com/catalog or drop a line to Sid and Shivram at hello@edfinity.com.
Runestone and Edfinity will be actively piloting our new offerings with several of you during the coming months, with full availability scheduled for Fall semester 2020. This is designed to be a community-centric effort and we invite your participation and assistance to any degree both in terms of conveying your needs to us, and in helping to build out an exceptional corpus of questions and unit tests. Please contact me with ideas and suggestions.
PreTeXt + Runestone: a New Partnership
I’ve just returned from an inspiring weeklong workshop on Interactive Assessments in Open Source Textbooks, hosted at the American Institute of Mathematics offices in San Jose. There were about 25 participants in the workshop, including, authors, instructors, developers, administrators, and people who wear combinations of those hats. Not only was the workshop inspiring, but the structure of the workshop encourages concrete progress, and I’m thrilled to say that we were able to demonstrate several aspects of a working integration between Runestone and PreTexT. Although our teams have been working in parallel for several years, it is clear that our goals are well aligned. The PreTeXt markup system provides over 50 different online resources to the math community including textbooks for Calculus, Linear Algebra, Discrete Math and much more. Both PreTeXt and Runestone have a lot to bring to a combined table and here are some of the ways that the entire community may benefit from an integrated system:
PreTeXt does not have a central hosting system like Runestone.Academy, nor does it collect the data for the kinds of instructor analytics that Runestone does.
While PreTeXt makes use of WebWork for interactive exercises WebWork does not have all of the different components that runestone has
PreTeXt has a different markup language than Runestone - based on XML - it is not complicated to write and less fussy than restructuredText in the use of whitespace in its Syntax
PreTeXt can produce documents in many different output formats including online as well as PDF, ePub, and in-progress work on Braille!
Serving a PreTeXt book with a Runestone Server integrating a WebWork question
During the week we were able to make PreTeXt output html that was compatible with several different Runestone components, including activecode, multiple choice, and short answer questions, the rest will follow over the coming months! This means that an author could write in PreTeXt and host a book on Runestone.academy. In fact were were able to demonstrate that we could already host existing PreTexT books, and with a little work we were able to capture student responses from WebWork problems and save them to the Runestone database.
In addition to the integration work, there was a lot of conversation and drawing of pictures and making of lists that revolved around topics that both Runestone and PreTeXt could benefit from. Some of it is great design work, some of it is a vision of how the future could look, But the answers to the questions will provide lots of work for the next few years.
How can we write and present instructor editions of textbooks? What kinds of resources should be included in an instructor edition and how best to expose those materials.
How can we build effective communities around the various textbooks we provide to encourage instructors to help and learn from each other?
What kinds of analytics would be useful for book authors? How can the data collected for authors be visualized effectively to help authors write more effective textbooks?
How can we write questions that can have many equivalent variations, to promote better student practice and learning.
To what extent should instructors be able to customize a textbook? How can we allow customizations without a dizzying and confusing proliferation of minor variations of the text?
How can we collect data to enhance student learning, while still respecting student’s privacy and respecting an increasing number of privacy laws?
What can we do to ensure that creators of open source educational resources get the proper credit towards tenure and promotion for work that has a wide impact? Sure a journal paper that is read by 6 other people is fine, but isn’t a book that educates thousands of students and moves pedagogy forward at least as valuable?
I’m not going to provide answers to those questions here as that work will be ongoing and we could use a lot of help. If you are interested in tackling any of these questions or just participating in the discussion, please let me know!
I also learned about the DOENET project happening across many universities with the project manager located at the University of Minnesota. This would be a longer term integration project that could address many of the privacy issues surrounding open source education materials. And since I’m only seven minutes away from DOENET central by bike, we decided to focus on other integration issues during this workshop, and get together after the conference to start this discussion.
In closing, I want to thank the organizers of this workshop – Jim Fowler, Mitch Keller, Matthew Leingang, and Oscar Levin for inviting me to participate. Thanks to the other participants who made the non-math guy feel extremely welcome! Thanks to Rob Beezer and David Farmer for inventing and writing PreTeXt and being so open to working on integration, even well into the evening after the workshop was done for the day.
I hope someone from the ACM will read this, and look into creating and funding workshops with a similar format. This was 100% a week well spent – Not something you can say after every conference! This workshop was a great beginning and I look forward to seeing what we can do together.
Privacy and Runestone
In this week’s update I want to tell you about some new account management features I added to Runestone, and give you some idea about how you could set up your class if you don’t want Runestone to capture any Personally Identifiable Information (PII) about your students.
Runestone collects a lot of data. I think it is all justified, and if you are an instructor you can gain some real insight into how your students are doing based on the data we collect and the reports we create for you. None of that requires any PII about your students. Yet we do collect some… what do we collect? and why?
Here is what we we ask for when a student registers:
username - this can be anything and it is easy for this to NOT contain PII
email - This is the only reliable way to do password recovery.
First Name and Last Name - this has always been just for the instructors so that when they are grading assignments or looking at analytic reports, they know who they are looking at.
Password - encrypted and stored securely.
Name of the course. The instructor has full control over this call it whatever you want.
Setting up a class with no PII
We will walk through two strategies for managing a Runestone Course that doesn’t store any PII about a student. The first strategy has the students register themselves following the instructions I’ll tell you about. The second strategy puts you in complete control and allows you to create all of the usernames for your students!
The way to use Runestone without PII is to ensure that your students choose a username that they remember but has nothing to do with their real identity. They can also put in fake first names and last names.
The real trick is how to manage password recovery Thats where this cool email alias trick comes in. Most email systems (including gmail) support the username+alias convention. That is you can add a + and an additional alias to your email address. For example runestone+support@gmail.com or runestone+info@gmail.com both of those emails will end up in the runestone mailbox, but I can also see the alias. We can use this feature so that students don’t have to enter their real email address, they enter yours plus their username as the alias. This means that you will end up fielding the password reset emails. If you are an email whiz you can set up some rules to forward these to the right student.
Experience has shown that having students register themselves and get all of those pieces correct is a losing battle. So why not put you in charge of registering them? Get out your favorite spreadsheet app and prepare a file with your students information.
You can upload a CSV file with the following format: One row for each student
username,email,first_name,last_name,password,course
In order to protect your students privacy you can create a spreadsheet like the following:
hogwarts-1,dumbledore+hogwarts1@gmail.com,F1,L1,owls4all,potions_fall19
hogwarts-2,dumbledore+hogwarts1@gmail.com,F2,L2,owls4all,potions_fall19
hogwarts-3,dumbledore+hogwarts1@gmail.com,F2,L2,owls4all,potions_fall19
Note
It is important that you check your CSV file and make sure there are NO SPACES before or after the commas!
From the Manage Students tab on the Instructor’s Page, you can now upload this CSV file using the interface.
Once you have uploaded the CSV, you can add some columns to the spreadsheet that contain the real student information to help you remember. You will probably learn quickly who your made up names correspond to, and can probably think of something anonymous that will help you remember who F1 L1 really is. On the first day of class you may want to hand out paper for the students with their username, and walk them through changing the password, or maybe you want to assign each student a unique password when you are making up the spreadsheet. I just would not recommend a scenario where all the students have the same password.
Of course all of this is optional. You can also create a csv file with usernames of your choice and include the real email addresses and real first / last names of your students, or you can have them register themselves as students have been doing on Runestone since 2012.
Password and Account Management
There is no doubt about it, password management is a headache for everyone. Runestone handles hundreds of password reset requests each week. But that does not always work. Some schools (inexplicably) block email from Runestone. Other schools run software that simply mangles the URL’s found in emails that look like they are password reset emails. This is understandable to some degree as Phishing scams are so common, and its important to protect the students. I’ve done everything I can do from my side to validate the runestone.academy email domain. But, other times students simply type in their email incorrectly.
When the email system for password resets do not work, I have now given instructors the ability to change the passwords for the students in their class. Hopefully, this will eliminate the emails asking me to manually reset passwords when the reset emails are now flowing. Runestone has gotten too big for me to deal with these requests, and I don’t have any minions for that.
Instructors cannot change their own passwords through this mechanism. who among us hasn’t walked away from the computer and left ourselves logged into Runestone? As an instructor, you will still have to use the normal email password reset mechanism. If your school blocks emails then I suggest you just get a free gmail address to use for this purpose.
One final note on account management. When you use the Remove Student button, you are not actually deleting the student. What I do is move the student out of your course so they do not clutter your gradebook or Student Progress page. They are moved into the public version of the course you based your own course on. This allows them to keep their username, as well as retrieve any of their work they did. If a student wants to truly delete themselve they do have that capability! From the user menu select edit profile and from there the student can really delete themselves and all of their work.
What other Data Does Runestone Collect
Here is a graph of all of the events that runestone has collected in the last 4 weeks.
As you can see that is a lot of activecode and a lot of pages loaded. Also over 1 million multiple choice questions answered! I would love to see more clickable area questions and more fill in the blanks. Some of our preliminary research shows that those kinds of questions are much more effective at getting students to review the content in order to get a correct answer, rather than simply clicking on different answers until they guess correctly for multiple choice. All of these events are captured and tied to a username. So the more anonymous your username the better as far as I’m concerned.
What I’m really interested in is figuring out how to apply machine learning algorithms to help identify students that may struggle, to help weed out bad questions, to help us write better questions, to help us order the textbook in different ways, to adapt to each student to help them achieve their maximum potential.
Runestone also collects the source code each time your student changes and runs the code in an activecode activity. This is to provide a timeline for the student as well as for yourself.
Make sure you explain the timeline to your students. Sometimes there is great fear in them about changing code that is working. Having this timeline present encourages them to experiment, knowing that they can always get back to a good working version.
For you the timeline can be an invaluable tool for talking through an assignment with a student. You can see their progress and if they make a wrong turn you can usually see that in the timeline and talk to them about why they decided to a particular direction.
The timeline is also available in the scratch activecode which some students like to use as a notebook for the course.
Grading Updates
Lots of updates to the Grading interface this week, as well as some important bug fixes. They are now live. The main things to look for include:
1. I have plugged a couple of holes in session handling, where students could mistakenly end up in the open fopp book instead of in the correct course.
2. In addition, if the student is not logged in, the progress bar at the bottom of the page will be gray instead of green. Another visual indicator that they are not logged in.
3. The grading interface has been redesigned into a new layout, and streamlined significantly. Important note – these changes are to make manual grading workflow less tedious. The autograder workflow is unchanged.
Now you only have one choice to make. Start with either a Chapter or an Assignment.
Questions for chapters are sorted alphabetically
Its easy to select all question or all students or both.
Chapters, Assignments, Questions, and students are all searchable. So if you have a class of 125 and need to regrade that one student it is easier to find them.
If an assignment has not had its grades released there is a bright red message to that effect that will remind you to press the button.
If you select multiple students you can navigate from score to score with the tab key and when you update the score it autosaves when you move to the next input field. You’ll know it has saved because it will turn light green.
If you like grading things one at a time, the old save and save/next buttons are still there.
Here is quick video tour of the new features:
Please don’t hesitate to leave a comment on other things I could tweak to make the workflow more clear or efficient.
Help Wanted - Make Autograding Awesome
What if I told you that just 10 minutes a day for one week between now and September 30th could improve your life dramatically for the entire semester?
Did I get your attention? How much time would you save if every programming problem in Runestone had unit tests so it could be auto graded? I know I would save me many hours! In addition my research has shown that students work harder, and end up with much better solutions when an exercise has a good set of unit tests. It seems like a win for everyone! When there is that positive affirmation that you are correct, students build confidence and feel better about doing the work. In addition, all the emails I get asking for solutions will go away because if all the unit tests pass people will be happy that they have succeeded.
Most of the programming exercises for Foundations of Python Programming, as well as those in How to Think Like a Computer Scientist lack good unit tests for the autograder. This is actually an easy problem to solve if we crowd source the solution. If all of us using Runestone in our classrooms write a unit test every day we could have 100% coverage.
Adding unit tests to nearly every activecode is an achievable goal, if we work together to write them. All the cool kids are doing crowdsourcing today, so why can’t we? Here is the best thing, I’ll take a set of tests for any activecode in (nearly) any format. Just send me the title of the book, the activecode ID (shown below the activecode) and a set of assertions, and I (or another volunteer) will get that information in the book. With the new architecture, as soon as I do a rebuild all of us will immediately benefit from the new tests. You can send me an email. You can post it in the comment section of this blog. You can create an issue on github Or you can post the message on our Slack Channel. (See the Support tab for the link to join).
The test you send me could be as simple as.
Dear Brad,
For activecode ac_2_5_blah we should test the following conditions
for starting input of 44 and 15
variable foo should have the value of 29
variable bar should have the value of "hello world"
The output should contain the word awesome
the source code should not include the word append
Sincerely,
<insert your name here>
Thats it, we’ll take it from there. You could also make Pull Request on github if you know how to do that.
You do not need to be an expert to do this. You could even make this an assignment for your more advanced students. I promise I won’t yell at you if you leave some corner case uncovered. Someone else will probably think of it. I, and a couple of other folks will act as the editors to make sure we have good tests that cover a reasonable set of cases for each problem.
What does a unit test look like?
Here is the ideal submission. (Again, I’ll take much less than this)
from unittest.gui import TestCaseGui
class MyTest(TestCaseGui):
def test_XXX(self):
assertEqual(variable, correct_value, feedback)
assertEqual(function_call(), correct_return_value, feedback)
...
myTest.main()
What about problems without functions?
No problem, you can make an assertion that a variable should be equal to a value just as easily as a function. Now, many problems are not stated precisely in the sense that we have not stated clearly. Store the result in “variableX”. That is easily changed as well,and you get the satisfaction of knowing that your variable name will be used by thousands of students.
You can also make assertions about the output as well as the assertions words that appear in the source code.
For example lets say you want to write a unit test to check that the ever popular “hello world” program works. We will use the getOuput() method to grab the output of the program. Its just a string so we can manipulate it and assert that substrings be in the output!
print("Hello World")
====
from unittest.gui import TestCaseGui
class myTests(TestCaseGui):
def testOne(self):
self.assertIn("Hello World", self.getOutput(), )
myTests().main()
Note that the ==== tells Runestone to hide the test code following the ==== from the student.
Looking for multiple answers in the output is easy to do with a number of assertions. So, this would be a good strategy for having them try several test cases. Even when using the input function!
Or, lets say you want the students to calculate the total of a list of numbers but you don’t want them to use the sum function, and you want the result to be stored in the variable total. We also have access to the source through the getEditorText method.
numbers = [1, 3, 5, 11, 25]
total = sum(numbers)
====
from unittest.gui import TestCaseGui
class myTests(TestCaseGui):
def testOne(self):
self.assertEqual(total, sum(numbers) )
self.assertNotIn("sum", self.getEditorText(), "Do not use the sum function")
myTests().main()
This would result in:
What if I have already written a unit test for problem xxx?
Even better! Just send me the unique id for the problem you created, and (if you know it) the id for the problem you started with.
I know we are all super busy, but with just a few minutes from each of us we can really make a huge dent in reducing our grading workload. Think about it this way: when you make the assignment, jot down a few ideas about what you are going to look for when the students hand in their work, or even write up the unit tests in an editor. If you get them to me, and we get them into the book before the deadline, all you have to do is make sure your students run them one time with the tests in place and your grading for that problem is done! I think this can pay for itself with a single assignment!
Thanks in advance for your participation!
Brad