PreTeXt Runestone Workshop
Happy July 4th Weekend! I am glad to be home after almost two weeks in Ann Arbor Michigan. But it was a great two weeks. Here are a few of the highlights.
The week of June 20 – 24 was the first joint workshop with the PreTeXt team. We had 28 attendees including authors, developers, educational researchers, and instructors representing the Runestone and PreTeXt projects. One of the main goals for the week was to give everyone a chance to get to know each other and learn more about the two projects.
The PreTeXt Runestone Partnership
After spending an entire global pandemic working on integration Rob Beezer and Brad Miller presented the integration of Runestone and PreTeXt. PreTeXt now has the ability to natively author most of the Runestone Interactive exercises including multiple choice, Parsons problems, Matching, clickable area, short answer, fill in the blank, activecode, and CodeLens! Rob has built a very nice annotated sample book (alternative link without source annotations) that illustrates these questions in action. Runestone Academy can now host books written in PreTeXt and fully supports using questions written in PreTeXt in our analytics and assignments.
Going forward PreTeXt will be the primary authoring language for books hosted on Runestone Academy. We will continue to research and develop new interactive components but we will not have to worry about inventing markup. If you are a current Runestone author I think you will really like the consistency and expressiveness of PreTeXt. It may take a bit to get used to the more verbose XML style of writing, but is well worth learning for future development. There are many advantages to using PreTeXt as an authoring language.
PreTeXt books are highly accessible
PreTeXt books that are interactive on the web can also be rendered in various static forms including pdf, ePub, Kindle, Braille, and more. We make two types of PDF, one is meant specifically for print/print-on-demand. If you want a bound copy of a book, you can get it!
PreTeXt books are highly structured, which will allow for more customization in the future. We spent a lot of time during the workshop talking about various ways we can make the textbooks more customizable.
One big topic of discussion at this workshop is how to grow the PreTeXt Runestone ecosystem. With our combined forces we need to do more in the following areas:
Developing champions for each textbook to develop community around each textbook.
Developing instructor materials for each book
Improving videos
Doing more blogging
Creating a new home page for Runestone Academy that is more informative about what we are.
Being visible at conferences and workshops.
New Titles Coming to Runestone Academy
Starting this summer we are converting many of our current books to use the PreTeXt markup language. The first of these is also the first textbook to be converted to Runestone. How to Think like a Computer Scientist: Interactive edition. We are finalizing the conversion, but I encourage you to take a look at what we have done so far on our development server I am super excited to see this all come together and you will see many other textbooks undergoing the conversion soon. In addition we will have a lot of other new titles available for the Fall.
Ordinary Differential Equations
Discrete Math: an Open Introduction
Active Calculus
Apex Calculus
Elementary Algebra
Intermediate Algebra: Functions and Graphs
Trigonometry
I am also pleased to announce that the classic Java Java Java: Object Oriented Problem Solving textbook is now available. Although this textbook was originally authored in LaTeX we have the scripts to convert to PreTeXt. If you have a open source textbook in LaTeX or practically any other format we can convert it to PreTeXt just like this one. Of course the conversion is only part of the work. What we talked a lot about at the workshop is how interactive textbooks that support active learning are structured quite differently than a traditional book. Our research shows that students jump straight to the questions or activities and only come back to read the text when they don’t understand the material. This makes for much more engaged readers.
New Features
One of the features of the workshop was a sharing of a lot of research done by the PreTeXt team as part of the UTMOST project sponsored by NSF. We also got a demo of a new feature of Runestone Academy that allows you to do Peer Instruction in your classroom. This peer instruction feature was the focus of the second workshop the week of June 27.
For Runestone authors it is also exciting to know that Sage and the R language are both supported natively by PreTeXt.
Starting a new book
It is easy to get started writing a book with PreTeXt using the PreTeXt command line tool. This tool allows you to quickly start a book, build books, and see updates to your book as you write. In addition there is also a great extension for Visual Studio Code!
We are working on an editor designed specifically for PreTeXt that will make it super easy for anyone to edit a PreTeXt textbook, more on that another day.
The future
One of the biggest goals for both PreTeXt and Runestone is to ensure that these projects thrive for many years into the future. This means growing the combined team, finding funding and many other things. If you are interested in getting involved in any way we would love to have you join the team. You don’t need to be an expert coder, you don’t even need to know how to code at all. We need willing proof readers, editors, testers of new features, community builders, bloggers, and authors. You don’t need to be a mathematician or a computer scientist either, great open source textbooks are a benefit to students studying any topic!
Summer 2022
Summer time is often when instructors of all kinds can help out with Runestone Development. Since June marks the beginning of summer I thought I would highlight what is on my list for development list for the next few months. If something sounds interesting to you then please get in touch!
PreTeXt
PreTeXt is now the official authoring language for Runestone. There are a host of reasons why this makes sense, and I’ve written about them before. It has taken some time but we have finally arrived in a place where it is a reality. I have converted our first Runestone book ever How to Think Like a Computer Scientist to PreTeXt and Rob Beezer has added markup to support our interactive elements. This is a huge step forward and we now begin a process of converting our current books. Like any conversion process it is not 100% so each book is going to take some manual work to perfect the conversion.
select questions
- toggle questions + A/B testing
I think the correct way to do this moving forward is to make toggles tasks and have Runestone show the tasks as dropdown
Convert select books to PreTeXt for Fall 2022
Build APEX Calculus for Runestone
Build ORCCA for Runestone
Build Yoshiwara’s Algebra book for Runestone
Finish up the SAGE Cell integration
Jaas - Jing as a service - Jing is a validator for PreTeXt
Make a PR to store traces with a hash of the input.
Peer + Grant Work
This work is funded by an NSF Grant for Barb Ericson at the University of Michigan.
Grading - 2 interaction Qs plus chat (maybe) - update autograder
Add async peer instruction page
Get tags working with question bank searching - what we really want is prereqs and reqs - add to web UI and make use of some taxonomy - someone please give me a taxonomy!
Automatically Create Parsons problems
This work is also funded by an NSF Grant for Barb Ericson!
Get Overcode working in python3 - this may also be a good task for Celery
Generate parson’s problems from the selected
Github Integration for Custom books
This work is funded by a grant from Elliot Houser and UT Austin.
Add simple interface to connect a Repository
Add simple interface to publish draft/published
Integrate the web hooks to auto build when pushed - this may be a separate server
Runestone Internal Projects
Updates to student progress page - replacing d3 stuff with pandas
- Allow instructors to download LARGE classes on their own - using a celery task with an email followup - maybe create a special role for trusted teachers / researchers to allow them access
celery could also be used to do build on demand of Runestone / PreTeXt books
Finish redoing the login process
prototype an author/researcher dashboard
Convert other books
Does PreTeXt support hidden pages? Somehow we will need a workaround for how barb has been adding problems to the database but not on a page that is linked to a TOC
Create a bibtex file with references and generate a web page of references from that.
Peer+ Webinar
Today, Barb Ericson and I hosted a webinar on Peer+. Peer+ is an NSF funded research project on using Peer Instruction in a Runestone eBook. Click the link to see a recording of the 1 hour zoom or feel free to just look at the slides
I won’t try to repeat the information here, but just emphasize that Peer Instruction is a well studied way to increase your students engagement and success in your class. Please check it out, and don’t hesitate to give us feedback on this exciting new feature.
The New New Academy
This is a big announcement that changes some things very little and some things a lot!
We have incorporated Runestone Academy and applied for 501(c)(3) status. Creating the new corporation is done and official. Eventually, Runestone Interactive LLC will cease to exist as a corporate entity. As of January 1, 2022 the server that you have come to know as runestone.academy is operating under Runestone Academy LTD.
Why the change?
The reason for the change is really quite simple. We want to keep Runestone free for all students. Period, end of story.
However, the reality is a bit different. In an ideal world every school that used Runestone would send us $10 per student. This would allow us to hire support staff and incentivise new authors to add more books to runestone for very little cost to the school and no cost to the students. But that does not happen. I know that many school districts do not have big budgets, and my goal has never been to “get rich from Runestone.” Don’t even get me started about the legal fees we would have to pay to have the “contracts” from school districts reviewed!!!
But We have big goals! I would love to see Runestone helping two million students a day instead of fifty thousand. To do that we need more authors, more books, more servers, more support people, and more programmers! I am greatful to all of the schools and teachers that send us support! Although I feel bad about teachers that send me money from their own pocket! I am from and education family and I know how little teachers at public schools and small colleges make!
So, what to do?
The more I researched this problem the more convinced I became that Runestone had to become a non-proft. This opens the door for us at many corporate and family foundations that care deeply about STEM education. We have several grant applications outstanding, but like everything else, this is a process. New groups have to prove themsevles, and that takes time. In the meantime we will be rolling out new programs that make it possible for individuals or schools to help us financially. More about that soon.
Academy Update
Thanks to everyone for your patience over the last three weeks! As a recap here is what has transpired.
I took Runestone Academy offline on Christmas day and started up my script to migrate the database to a new machine dedicated to running Postgresql. That script took just over 25 hours to complete. It modernizes the database and adds many new constraints to make sure that when we store something in the database it is good and consistent and valid.
On January 27 I brought up two “worker bee” servers and our load balancer. This allows me to add as many “worker bees” as we may need to serve the traffic going to runestone. I changed our DNS entry to point “runestone.academy” at the new load balancer. Unfortunately I did not delete a previous DNS entry so now some people were getting the old address and some the new. Chaos ensued for a few days.
By Monday January 3rd I was patting myself on the back and thinking “this is going better than I thought” LOL – Many problems – to numerous to count ensued as soon as classes actually started again.
LTI - or what I call the “community supported feature” was just supposed to work. But did not. Thankfully I think that as of yesterday it is now working great!
Today was the first day since December 27 that I have not had a new bug report. I know there are still problems out there, but at this point they are outliers not the norm. That does not mean I don’t care, I do, but it does mean that I need a lot of detailed information to try to figure out what is wrong. Major widespread problems are easy to figure out because I hear from hundreds of people. I’m not hearing from 100’s anymore!! So details are important.
If your students are having trouble
Logout - clear browsing history, and log in.
Try incogito mode
Open up the Javascript console on their browser and send me everything! No detail is too small. And Its not your fault! at this point I am trying my best to debug the odd cases. I am even willing to do a zoom with you and your students to debug the problem.
I cannot emphasize how useful it is to do #3! So much of what happens in Runestone is delegated to the browser. If we did not delegate this work to the browser we could never scale to as many students as we have!
Thanks for sticking with us through this transition. Major shifts like this are always painful, and I know we are not through yet. But this new architecture sets us up for a very bright future!
Academy Downtime
Runestone Academy is undergoing some major upgrades this holiday season! Unfortunately these changes will result in some downtime. Here is our schedule of events so you know what to expect in the coming weeks.
December 25 - 26 Runestone will be offline as we move our database server to its own machine. In order to ensure that we don’t lose any data we need to take all of the books offline while we move the database. This is also a great time for us to do some maintainance on our data in preparation for our new servers.
December 26-27 Bring Runestone.academy back online. In this initial period the old servers will come back up but use the database in its own location. There may be some unexpected crashes during this period as we have added a number of protections to the database to keep future data problems.
December 27–January 1 Migrate away from the old server and onto the new servers. If all goes well this is mostly just some DNS changes, but it will still likely cause the appearance of some downtime as you may not reach the new server and / or the old server may look like its certificate is expired.
January 3 and onward Back to business as usual. Although there are likely to be some hiccups as traffic picks up with students and teachers getting back to school.
A Few Details
For those that are interested, this change is pretty big. Here is what Runestone has looked like in the past:
Runestone.academy runs the web2py application server as well as the database server (postgresql). This server has been in service since 2017. I haven’t had to reboot it for 695 days as of this writing! It is definitely showing its age, and can definitely not keep up with all of the traffic. Our stopgap measure to alleviate the pressure on this machine was to add csawesome.runestone.academy and route all traffic for the csawesome book to that server. It still used the database on academy. And it used most of the cpu power of the five jobe servers that compile over 100,000 Java programs each day. If it looks like this is a “Rube Goldberg” contraption its because it is.
For years we have wanted to move to a new architecture. And this change is our first step in that direction. This new architecture is all based around Docker containers and gives us the ability to add more capacity in just minutes. It will also let us scale back during the less busy summer months, thus saving us money as well.
All traffic will now go to runestone.academy. It will be routed to one of N docker containers that run the old server which is now just for the instructor dashboard. Page requests for books will be handled by our brand new bookserver. This new server is built using a very modern web framework and will have a ton of benefits for developing new features going forward. Each container also houses a JOBE server for handling java and other compiled languages not supported in the browser.
Warning
Any big change like this is bound to come with some setbacks. I’m sure there are things around the edges and in the corners of our new server that will break over the first couple of months. Please be patient, and please please report problems on our github issues page Don’t assume that someone else will report a problem or that it is your own fault.
It will also take us some time to tweak the performance of this new setup. We have had almost five years to squeeze as much performance out of the old system as we could. It will take some time to get the most out of this new one as well. So again please be patient and let us know of any particular bottlenecks.
Please Support Runestone
As long as I have your attention, and we are at that end of the year time when we think about supporting the organizations that matter to us, please consider a small donation to keep runestone solvent. You can donate directly on PayPal Thanks!
Enjoy the holidays!
Brad
Annual WeBWorK report
As we near the end of the year, it feels like a good time to report on Runestone’s foray into WeBWorK hosting. In Fall 2020, Runestone began this service. Three factors contributed to this:
Runestone had just set up its own WeBWorK server to support PreTeXt books that use WeBWorK.
The MAA hosting service was coming to a close, and many users would soon have no hosting service to turn to.
The COVID pandemic was on the upswing, and institutions everywhere were looking for new (to them) online tools to support remote learning.
So it seemed like a useful idea for Runestone to offer a WeBWorK hosting service. For that first Fall 2020 semester, hosting was free while we assessed the cloud computing and customer service demands. Since January 2021, hosting incurs a $100 fee per course.
Facts and Figures
Over the terms and semesters of 2021, the service has supported:
62 institutions in 8 countries (55 higher ed and 7 secondary)
117 individual sections
Our revenue from this service was $10,800. For transparency, here is where this money has gone. These numbers are preliminary, as the year is not yet over. But I expect they won’t differ by a significant amount. - $5400 to Alex for work hours - $1493 operational expenses - $1076 Runestone paying taxes - $1365 transferred to The WeBWorK Project - $1365 retained by Runestone
The last two items are particularly important. These amounts may be small, but they contribute to a future of sustainable support for WeBWorK and Runestone. Each project is an open source software development, and relies on “tens of thousands of dollars” in annual development. That is in quotes because for the most part, the work is done by unpaid volunteers. That’s not a sustainable practice, and we lose good contributors to burnout. Real funding will help the developers feel appreciated, and in some cases buy out a chunk of their day job duties (say, one course release) and provide real time for their efforts.
Moving Forward
My goal is to grow each of those last two numbers up to $4000 for next year’s report. I think this can be accomplished considering:
Growth in the number of subscriptions. Already from Spring 2021 to Fall 2021 there was a lot of growth. So even if there is no more growth, 2022 will have more revenue than 2021. But it is reasonable to expect continued modest growth.
Raise the average cost per course to around $120. The additional money would go towards WeBWorK and Runestone, not labor costs or operational costs. My hope is that with transparency like this report, a price increase like this is not ill-received by subscribers. The increase is to support the software itself.
Offer and highlight other avenues for contributing. For example, a higher tier course subscription option.
There are other factors to consider for a new pricing scheme.
Should secondary schools pay the same as higher ed institutions? It is typically more difficult for a secondary school instructor to have an expense like this approved. In general, the budget is tighter and there is more red tape.
Some institutions have a dozen or more courses, and some have only one. Should there be some sort of departmental discount?
Some courses have a dozen students doing 5 exercises once per week. Some courses have several hundred students doing 30 exercises twice per week. Should a pricing scheme address that discrepancy? Certainly the high volume courses are the ones responsible when we need to upgrade our cloud computing resources.
The Runestone Team will study the options soon and publish a new pricing scheme that will come into effect Fall 2022. (There will be no change in the $100 fee per course through summer 2022.)
Other Significance
In the course of managing a WeBWorK server across multiple institutions, a few WeBWorK bugs were naturally exposed and have been addressed. Typically, something to do with a course’s time zone differing from the server’s time zone. Also, with so many users having one clear contact to report a perceived bug, this has been a good feedback collection device. And then simply to support multiple institutions using the same server, a number of new features were added. Overall I credit around two dozen improvements to WeBWorK 2.16 and (upcoming) 2.17 to this Runestone hosting project.
Thank you to all of our subscribers. You are supporting the continued improvement of two significant open-source apps for higher ed. May these projects continue to grow and serve you better.
Alex Jordan
On Data Privacy Agreements
An Open Letter
Rant Alert
Before I get into the heart of this I will apologize in advance if this sounds a little like a rant. I started this post on a day when I got three different reqeusts to sign contracts! I am trying to clarify my position, open a dialog, and provide you some alternatives for using Runestone without needing a long contract.
Dear School Districts,
I run Runestone Academy as a “labor of love.” For me it is a great way of paying back all those who have helped me be successful over the years. The mission of Runestone is to democratize textbooks for the 21st century. One implication of that mission is that everyone who wants to learn computer science should have access to free, high quality materials.
I’m not trying to turn Runestone into a major corporation or even make a profit. If I can run it as a small business that keeps me from becoming a greeter at Wal-Mart that is even better! I’m perfectly transparent about the finances, please take the time to read about that in our Annual Report It also does a nice job of clarifying our mission.
Districts, you are driving me crazy with your student data privacy agreements! Do you even understand what you are asking for? I understand that you must follow laws that have been thrust upon you by your well meaning state legislators, I understand that student data privacy is a good thing. I understand that you don’t want some corporation making money by selling your students PII for fun and profit. I agree! Read my Privacy Policy and our Terms of Service We are on the same side.
But here is the problem:
As a free and open source project we don’t have lawyers, nor do we have the budget to pay for them as needed. I am definitely not a lawyer, and frankly Its very risky for me to spend my time pretending to be one to read your agreements.
As a free and open source project I work on this because I love to write educational materials that help students learn and I love to code systems like Runestone that help you and your students teach and learn more effectively. This does not include spending my time filling out attachements A, B, C and H of yet another data privacy agreement.
As a free and open source project we don’t have any money to promise to reimburse you for anything if something goes wrong.
Approximately every district in every state has a different agreement!
As a professor of 15 years I am very aware of FERPA and all that it entails. But seriously here is what I am asked to agree to on a regular basis. This is taken directly from contract sent to me by a school district like yours.
The Contractor warrants that the service it will provide to District is fully compliant with the following regulations:
Children’s Online Privacy and Protection Act (COPPA)
Family Educational Rights and Privacy Act (FERPA)
Health Insurance Portability and Accountability Act (HIPAA)
Payment Card Industry Data Security Standards (PCI-DSS)
Student Data Privacy Pledge
And any changes to the preceding that may occur from time to time.
The vendor hosting data center meets the following standards (proof must be provided if PII is stored at rest in the vendor’s data center):
ISO27001:2005
SOC2
SSAE 18
Sometimes I swear you and your lawyers just Google everything related to privacy and add it to a laundry list for us to figure out. Honestly what does HIPPA have to do with computer science? I asked a lawyer friend about HIPPA compliance and he laughed and said that only a handful of people in the country even know what that means.
As a non-lawyer computer scientist I have learned some heuristics that help me weed through these agreements in record time. All I need to do is search for the words “indemnify” and “reimburse” as in:
If requested by the District, Provider shall reimburse District for costs incurred to notify parents/families of a breach not originating from District’s use of the Service.
The Provider shall indemnify and hold harmless the District from and against any loss, claim, cost (including attorneys’ fees) or damage of any nature arising from or in connection with the breach by the Provider or any of its officers, directors, employees, agents or representatives of the obligations of the Provider’s or its Authorized Representatives under this provision or under a Confidentiality Agreement, as the case may be.
Really!? If Equifax, major banks, and pipeline companies cannot always prevent hacks! How are we supposed to be any better? We do our best, try to follow good practices and take all reasonable precautions, but you now want me to promise to pay you back for a service that you are using for FREE. If I find a clause like that I will ask you to remove it. I simply cannot enter into a contract that obligates Runestone to pay you back.
Trying to guilt me into signing something will not work. “If you don’t sign this then our students will not be able to use your materials.” Yes I want your students to use the materials, its why I do this, but I don’t make anything from your students, Its not like a for profit corporation where if I don’t sign you don’t pay me. In fact every district that uses Runestone just adds to the costs of running Runestone. More students means more servers means more money we have to pay to keep running the service. I am not going to put the project at risk just to make your lawyers happy.
Every minute I spend trying to deal with your red tape is a minute a don’t get to spend making Runestone better or writing new materials to help your students learn.
Solutions
First, everything on Runestone is open source and free for you to use however you see fit. This includes every book and the Runestone server that supports it! This allows for several solutions.
You can run your own Runestone Server and serve any of the books you want on your very own server. This totally eliminates the need for you to use Runestone Academy and keeps you 100% in control of your students data. This was actually the model I envisioned for Runestone in the beginning, but I quickly learned that many people do not have the time or the skill to maintain their own server, and so to help those people Runestone Academy was born. No good deed goes unpunished as they say.
You and your students can read and use any and all of the books on Runestone Academy without creating an account. Ok, yes this makes life a lot harder for teachers to gather their students homework and grade it, but it is 100% an option. I am not trying to force anyone to create an account so I can gather PII to sell!
You could agree to send Runestone some money so that I can have your agreement properly reviewed by a lawyer at around $350 / hour. Lets call it $1200. In that case I would be happy to agree to refund that entire $1200 as reimbursement for a data breach. I’m happy to put that in writing and send you an invoice.
You, as a teacher, can register all of your students anonymously with NO PII. See this post for instructions on how to do this. I would be happy if everyone who sent me contract would just do this instead. I have no interest in knowing anything about your students, the only reason I collect names and email addresses is to deal with password resets and to make life easier for instructors when they are grading assignments. This solution makes it a bit less convenient, but does so without any PII.
In the back of my head I have an addition to Runestone that would allow everyone to use option 4, but allow instructors to keep a mapping in their browser (NOT on Runestone) that would let them see who the anonymous students really are. That is just a gleam in my eye at this point, no code has been written, so please don’t ask when it will be ready.
We Are in this Together
I firmly believe that our goals are aligned! I want to provide great free resources to help students learn, and you as school districts with limited budgets want to use great free resources to help your students learn. I am not the enemy! I have signed the student data privacy pledge, I have published a privacy policy. I sincerely care about student privacy!
But, I believe these laws put projects like Runestone at a real disadvantage. We do not have the resources to hire lawyers to review and negotiate all of these contracts. We are not big enough to make you sign our own. So we need to work together and treat each other with respect and maybe engage in a little give an take. It is a win win! If you don’t use Runestone are you going to pay a big publisher $100 per student every semester?
If you have other suggestions for how we can make this work please feel free to reach out to me, I am open to hearing ideas and working on alternatives that do not create more legal hassle for any of us!
Thanks for reading!
Sincerely,
Brad Miller
Professor Emeritus, Luther College
Founder, Runestone Interactive
Runestone Roadmap 2021
There is a lot going on in the Runestone world right now and I wanted to let you in on the current roadmap for Runestone development. I’m doing it now as summer is on the horizon and I know for many of you summer is the one time of the year that you may be able to help with development! More on that later. Here is what is on the Horizon for Runestone.
Roadmap Overview
BookServer
Although the web2py framework has served us well since 2011, I have wanted for a long time to rewrite the core book server in Runestone using modern web frameworks and technologies. This will allow us to scale Runestone more elegantly and save us money as well! The Instructor interface, which gets 10% of the traffic that the book server does will stay on web2py until I or someone else finds the time to port it as well. They will continue to work together just fine as they will share a common database. The BookServer project is using a fully async framework called FastAPI as well as SQLAlchemy for the database work. It will be tightly integrated with the Runestone Components so that you can easily spin up a server to serve a Runestone book on your own infrastructure, With access to all your data in a sqlite database. Production on Runestone Academy or another large installation will still use Postgresql. This is the highest priority for me at the moment as I don’t want to do any further student facing development on the old server.
Funded Projects
There are several high priority projects that are being funded by NSF money through colleagues like Barbara Ericson at the University of Michigan, or the SPLICE project. Others may be on the way…
Support for Peer Instruction
The peer instruction project is part of Barbara Ericson’s research at the University of Michigan and has a two year timeline.
1. Allow Runestone to ingest question banks from various projects so that we increase the number and quality of questions available for instructors to assign.
2. Integrate a discussion system into Runestone and the assignment system to allow students to first answer questions and then discuss them with each other synchronously or asynchronously before they answer the question again. This is the heart of the Peer instruction model.
3. Analytics for selecting questions suitable for peer instruction.
An instructor interface to allow an instructor to see and share aggregate information about how students are answering a particular question.
Support for automatically pairing students for peer instruction.
Publishing Anonymous Runestone Data
This is an ongoing project funded by the SPLICE project. If you are interested in analytics or machine learning and how we might improve the learning experience by applying machine learning to Runestone books I have made available several datasets using the DataShop system at Carnegie Mellon University. Contact me for access to the data.
Support for Large Online Courses
This is a project that is adding support for courses on Coursera using Runestone as the underlying textbook and backend. And includes a number of miscellaneous tasks.
Supporting the new toggle question. Given a particular coding problem allow a student to answer a series of hint problems (maybe even a parsons problem) to guide students that are struggling to a solution. Conversely many students would rather write code than drag parsons blocks around so in that case allow the student to simply write the code.
Improve the performance of the autograder and the student progress dashboard for large classes.
Modify the assignment page interface to allow for a group of students to work together on an assignment.
Smaller Projects
Change the Runestone Onboarding (Registration) Process
Separate registration from adding a course. The flow should look more like the following:
Register for Runestone Academy.
Are you an instructor?
Provide a bit more information so that we can verify that you are.
Here is an overview of the books we offer. Select one to create your course.
Guide you through the course creation process and give you some resources to make the most of Runestone.
Are you a student?
If you are an independent learner here are some open courses you can join.
If you are part of a course here is a nice interface to find the course by name and register for it.
Support for Custom Textbooks
This is a feature than lots of people ask for, but I’ve never found a scalable way to implement it… Until one day as I was registering Runestone on “Read the Docs” it occurred to me that the readthedocs.org model would work perfectly well for Runestone!
This development effort would add webhooks for git and Runestone to allow an instructor to Fork any of the Runestone textbooks (or create their own) and link it to Runestone.Academy for hosting. When changes are pushed to the forked textbook Runestone will automatically rebuild and publish the book as a draft. The instructor can then choose when to upgrade the draft to published using the instructor interface.
Even for instructors that do not want to get deeply into restructuredText or python programming this would allow a book to be customized using github’s web based editor. With rebuilds run automatically it would allow authoring a book with no setup.
A somewhat related part of this project is to update our library page to
make it database driven. Right now we simply walk the directory of books
and rely on the presence of a conf.py file that we import. This is
error prone and does not allow us to distinguish books that should
remain private versus books that are available to create classes from,
etc. This part of the project would make the library page as well as the
course creation page database driven.
Real time UPDATE I just found out this project will be funded through a grant from Elliott Hauser at University of Texas at Austin !! Elliott is the founder of Trinket and a long time collaborator on the Skulpt Project used to power Python in the browser.
Editorial Interface for Editing and Updating Questions
As Runestone has grown and as more instructors gain experience and confidence in using Runestone the size of our question bank has grown. However my ability to manage all of these questions is far less than what is needed. This project would build out an interface for volunteer editors to clean up and remove bad questions as well as make slight editorial updates to typos or unclear instructions.
Tag all Questions within a Taxonomy of Competencies
With the addition of the exam generator to Runestone we have a real need to create a way of tagging questions with the various competencies that they exercise or test for. This could be related to the editorial interface, but another great suggestion is to crowd source this and have students participate. This project would enhance the question interface to allow students to tag/categorize key concepts for each question. Ultimately editors may have to make the final decision, but it would be a rich dataset to see what students think.
Enhance the Web Interface to allow Instructors to Publish Projects or Labs
Although it is already possible to create coding exercises a number of books are more project oriented. It would be great to allow instructors to publish units of work that are larger than a single exercise but smaller than an entire book. More like a project size unit that may consist of text plus a number of exercises.
Convert Runestone output to XML (PreTeXt)
My work with the PreTeXt project has brought several math books to Runestone. PreTeXt is an authoring system that is primarily used for math books but also some others. It has the advantage of allowing you to write once and choose several output formats. Including PDFs, this is something that I get asked about all the time. My usual answer is why would you want a pdf of an interactive book? That just eliminates the interactivity! But the truth is there are lots of situations where people want or need a pdf version in addition to the interactive version. This is work that I’ll be at least starting on at a workshop in late June hosted by the PreText Project.
Upgrade LTI to Version 1.3
Currently Runestone supports LTI 1.1, the future is 1.3. Anyone who would like to take ownership of Runestone’s LTI integration is more than welcome to do so. Since I do not use an LMS as part of my daily work anymore I have little motivation to work on this. More work is also needed to verify and document how to use LTI with other LMS systems (besides Canvas and Moodle)
Template Questions
We have a prototype of a fill in the blank questions that would allow you to write a question where every student would get the question but it would be slightly different. For example you might want to write a question like What is X + Y? where each student would get a different X and Y chosen from a range of integers. The question would be smart enough to use a computed answer based on the X and Y given to each student. We would also like to expand this concept to multiple choice questions. Not only is this great for exams and keeping students from cheating but it also gives the students more practice as they could see a different version of the same problem each time they refresh the page.
Blockly on Runestone
Thanks to the authors of the amazing Mobile CSP curriculum and book (coming to Runestone for Fall 2021) and the hard work of Ralph Morelli we will have a great addition to the runestone library, and the ability to use blockly in your textbooks. I think this is awesome for beginners and I’ve always wanted to update The How to Think Like a Computer Scientist (HTTLACS) book to illustrate each new programming construct in Blockly first. I think this is a really cool way to give students a visual understanding of loops and conditionals before they see the syntax.
As is usually the case with all things Runestone we start small, and then iterate, so if you are into blockly and would like to help evolve this new component (which goes by quizly in the source) we would love to have your help.
Fixing Bugs and Even Smaller Projects
Many small coding projects are available on Runestone and are identified as issues. These issues are all typically tagged with the “Good First Issue” and or “Help Wanted” tags. I try to identify these projects this way because they are good starter projects that can get you into the Runestone system. See
Runestone Server Projects - Good First Issue
Runestone Component Projects - Good First Issue
Video Production Anyone?
The YouTube videos that accompany the HTTLACS book are very popular on YouTube, but I consistently get complaints about their resolution. Unfortunately the original recordings are 8 years old at least, and have been lost through many migrations of computers. They also feature version 0.1 of Runestone so that pages don’t match at all anymore. My new years resolution for 2021 was to work on one of these a week. So far we are at week 17 of 2021 and I have redone exactly 0 videos. If you would like to try your hand at re-recording any of these I would love it! You can just take the transcript of an existing video and work your way through it.
How You Can Help
Runestone is an Open Source project. Everything on Runestone is freely available as part of our Github Organization. Summer may be a good time for you to help hack on something, test something, or write something! I would be happy to have help on any and all projects. Funded or not it does not matter to me. I only point out the funded projects to show how awesome it is that some big partners have put their faith and money behind Runestone to keep it sustainable.
You do not have to be an expert coder to help out. We welcome people of all abilities to our community and would be grateful for your help.
Another way you can provide coding help to Runestone is to consider us as an option for a student capstone project. We have had great success in working with students at Berea college and other colleges that have adopted Runestone for their capstone course. Many of the projects identified above would be great as capstone projects for a group of students. The students really love, especially if they have benefitted from using a Runestone book as part of their education. This is a pathway for them to give back to a project that has helped them.
Another way you can help is to talk to encourage your department to budget some money to help support Runestone. I’ve heard from a number of instructors this spring asking about this, and I’m very grateful. We are happy to send an invoice for whatever amount you decide. Other departments have decided to charge a lab fee of $10.00 for the course, and have designated part or all of the per student lab fee to go to Runestone. We are happy to work with you on billing and payment!
Annual Report 2020
Happy 2021! Quite a few people have asked through various channels the last few weeks about how Runestone is doing. So it made sense to answer that question in the form of an annual report. The mission of Runestone Interactive LLC is to “Democratize textbooks for the 21st century”. There are two important components to that mission:
Democratizing Textbooks — what that means is that ALL students should have equal access — for free — to high quality books. Expanding outward a bit I think the ideals of democracy also encompass our committment to student privacy. If it was possible to run Runestone with zero student personally identifiable information, I would do it.
What are textbooks for the 21st Century? This was really the genesis of Runestone, that textbooks, especially computer science textbooks ought to encourage active learning and therefore BE interactive. I’m really proud that not only are Runestone’s textbooks highly interactive, but many of our features have PhD quality research behind them. I can trace six different contributors to Runestone who have used Runestone or developed features for Runestone as part of their PhD Work.
I think there is more to come here as we learn how to apply machine learning to the data we have been collecting over the years. Imagine if the textbook could intervene and give the student intelligent hints? Or what if the textbook could “see” the early warning signs of a student about to lose their way in the course? There is much to aspire to in creating textbooks for the 21st century!
In the rest of this report I want to give you an overview of a few of our accomplishments during 2020, particularly in the areas of growth and financial viability. I’ll try to restrain myself and not dive into all the Python and Javascript code I’ve written this year!
Overview
2020 was an exciting and good year for Runestone along many fronts.
We experienced huge user growth, partly due to the pandemic, and partly to new books with great support and large reader bases.
We hosted nine new (to Runestone) textbooks.
We hosted five Math textbooks, our first outside of CS, thanks to a partnership with the PreTeXt project.
We made strides toward making Runestone sustainable, experimenting with ad serving and ramping up our efforts in getting support from institutions and individuals.
We hosted over 200 instructors in our first virtual workshop, held in June.
Several colleges have used Runestone as an integral part of their capstone project experience for students. This is an amazing way for students who have learned from the Runestone textbooks to give back — by helping with development.
We put in place legal Privacy Policy, Terms of Service, and a formal information security policy.
The University of Michigan funded a major development effort to create an exam builder. This is still in Beta but you’ll hear more about this in 2021.
Key Metrics
Registered Users
The graph below shows the cumulative number of registered users on Runestone Academy starting in January of 2019. You can see the sharp uptick in the number from about mid august of 2020 to late september of 2020. The number of users is very much tied to the rhythms of the school year as we see big numbers at the typical starts of each term. However as we serve students around the world this is somewhat moderated by the fact that in the southern hemisphere the cycles of the school year are the opposite of the northern. Much of the growth in fall registrations is due to the CSAwesome curriculum and the many high schools that have adopted that for their AP CSA courses.
graph of user growth over the last two years
You can see the cyclic nature more clearly in the following graph that illustrates the number of new courses created each week. Particularly interesting in this graph is the spike of new courses created in March 2020, around the time of widespread stay at home orders in the USA.
Courses
Graph of the number of new courses starting each week
Courses for each Book
The majority of the Runestone traffic comes from a few of our books. The CSAwesome book has been a huge hit, taking the number one spot from the How to Think Like a Computer Scientist (thinkcspy) book in 2020. The Foundations of Python Programming book continues to show good adoptions as an alternative to thinkcspy in the Python area. The Webfundamentals book, which has continue to evolve since I wrote it for my own course about five years ago continues to do well. Finally the How to Think Like a Data Scientist book, written with the support of Google, is seeing some traction outside of the original group of 20 schools that were guinea pigs for that book.
One of the challenges is that we have great materials on Runestone Academy but have always relied on word of mouth as our marketing and advertising strategy. I think many of our books are poised for big growth if we can figure out effective ways to spread the word. One of the reasons that CSAwesome is so popular is because of the training and professional development opportunities provided by the CSAwesome team that brings in hundreds or thousands of teachers each summer and introduces them to the materials.
Graph of all courses created in 2020 by the book used
Graph of the number of students using each book in 2020
Page Views
The number of registered users and formal courses tell only about half of the story for Runestone. Roughly 50% of the traffic we experience is from search engine traffic. Daily we see an over 300,000 page views on runestone.academy, or over 2 million page views a week. According to Alexa Internet, that puts Runestone.academy in the top 10,000 websites in the United States. And the top 1% of all websites in ithe world! Hard to believe.
As you can see, the weekends are dramatically lower traffic as this graph over the last two weeks confirms what we have all long suspected about our students work habits. It is also why I update all the books and push bug fixes on saturday mornings.
Total pages views each day for the last two weeks (partial day for Jan 27).
Financials
In May of 2018 I took an early retirement package from Luther College to work full time on Runestone. This was a big leap of faith, but softened by the package offered by Luther. After making that leap I was very fortunate to have the opportunity to get a contract with Google’s Engineering Education group that allowed me to work on the Runestone platform and develop content for one of their initiatives. That contract (originally for five months) stretched until June of 2020. So revenue was not really a concern until June of 2020. Since then we have taken some steps to meet the goals of making Runestone sustainable, and of allowing me to continue to devote full time to working and supporting all of the students, teachers and authors using this platform.
Expenses
On the expense side we had a total of $18,000 in expenses. We can break things down into the following categories:
Hosting: $5,000
Health Insurance: $6,000
Financial service charges: $750
Legal Fees: $450
Other Business Expenses: $5800
Hosting charges have more than doubled for 2020 due to the increased traffic to the books as well as the increased use of languages that require service side support — Java and C++. The legal Fees were largely for reviewing our privacy policy and terms of service, but increasingly school districts are asking for us to sign legal contracts in order for their teachers to use Runestone. Many times those agreements run 20+ pages long and are hard to read for a non-lawyer. Most of them are in my view quite unfair, and expect Runestone to promise to reimburse the school district for any expenses they incur in the case of a data breach. All for a service they are receiving for free!
Revenue
Given that we really started to focus on Revenue in mid-summer I’m very pleased with our progress on this front. We had a total of $65,000 in revenue. Our major sources of revenue break down as follows:
Support from individuals through Paypal, Venmo, Patreon: $10,200
Support from Colleges and School Districts: $20,000
Grant and Consulting Income: $31,000
Advertising Income: $3,500
I’m really pleased with how this is working out. The difference between Revenue and Expenses leaves about $47,000. As I easily put in 40+ hours a week coding, maintainnig, and supporting instructors and students Runestone, that does not work out to a great hourly salary. It is less than half of what I would like it to be, but it is a great start! It is also a labor of love. The support from researchers at Elms college, and the University of Michigan has been wonderful. The $100-$300 level of support from many high schools and college departments obviously adds up and I hope that will continue to grow. It is pretty easy to project that our Advertising income will grow quite a lot in the next year as the number for 2020 only includes the fall quarter. As a reminder, Advertisments are ONLY shown to users who are not logged in, which means that the vast majority of ads are shown to people who arrive on Runestone through some kind of search.
If you are an instructor and can encourage your students to support Runestone or can persuade your department to provide some support that would be awesome! If you are a educational researcher and would like to include Runestone in a grant, lets talk!
Many people have suggested that I simply charge every student $10 or $20. However, as I mentioned in a blog post earlier this year one of the “True North” guiding principles for Runestone is to keep textbooks free for students. It makes it harder for us to stay viable but I believe with your help we can do it.
Challenges and Goals for 2021
The challenges for 2021 are many and varied
I am currently in the midst of rewriting the core Runestone Server. We cannot overcome our scale issues by simply spending more on hosting fees. Runestone is built on a foundation that is over 10 years old. Moving to a modern web framework requires a lot of work, but will pay off.
Growing overhead for Privacy and other legal agreements. I am 100% in favor of student privacy, and as I’ve repeated many many times, I will never sell student data, I will never give away any data that risks exposing any student’s identity. But there has to be a solution to dealing with schools that want me to sign onorous legal agreements for a free service.
Growing the community. If there is one thing I have learned in the last two years it is the importance of building community around the Runestone books. The huge growth for the CSAwesome books is partly due to great content and partly due to the amazing community that Beryl Hoffman and Barbara Ericson have built around that book. If you have some time and energy to devote to creating a community of practice around any other book I would be happy to have your help.
Continue to grow revenue. I can only dream of the day when our revenue brings in enough money to hire a second full time developer or a support person. If you have an interest in Search Engine Optimization, and would like to help us out in an ethical way I would love to hear from you. One additional project that I’m committed to is to remove the links to twitter, Facebook and other social media platforms. Although, in the beginning I thought this was a great way to increase the visibility of Runestone through social media, it seems that now it is mostly a benefit to their advertising platforms. Time fo those links to go.
I’m happy to say that the National Science Foundation has awarded an IUSE grant to Barbara Ericson for Peer+, a system to support both in-person and remote Peer Instruction in Runestone ebooks. In Peer Instruction students vote for an answer to a hard multiple-choice question individually, then discuss it with a peer, and then vote again. The instructor then displays the results of both votes and discusses the answers and misconceptions. We will mine historical data from the current ebooks to identify good questions for Peer Instruction and also add questions from public question banks. We will test the effect of Peer+ on learning, student satisfaction, and student retention.
I would like to develop a formal advisory board for Runestone along with a succession plan in the event that I am eaten by a bear on one of our camping trips, or run over by a car on one of my many bike outings! We already have an informal advisory group consisting of a group of Authors that meet on zoom every couple of weeks. Namely Bryan Jones of Mississippi State, Beryl Hoffman of Elms College, Barbara Ericson of University of Michigan, Jan Pearce of Berea College, and James Juett University of Michigan. Others pop in from time to time but I am very grateful for this group that provides regular feedback and wisdom! In addition if you are an operational person and would have an interest in helping Runestone by being the go to person for keeping the servers up and Running or have an interest in becoming a Docker/Kubernetes expert I would be only too happy to give up the responsibility for care and feeding of our servers.
As we begin 2021 I am feeling very hopeful. Although most days I feel like I’m a juggler juggling one too many balls, I think Runestone Interactive is poised to live up to our big goals! That cannot happen without your help. If you like to code, there are lots of easy starter tasks to take on to get you started. If you like to interact with other instructors, how about adopting a textbook and helping to build community for that book? If you like to write or make videos, how about helping us with our documentation? If you have some influence or some spare change laying around your department, how about supporting in future of Runestone?
We have a great mission — Democratizing textbooks for the 21st century! Help us achieve it.