Adding the Power of Open Online Annotation to Older Web Browsers
Wrap-Up Report on Hypothes.is’ Open Annotation Fund Grant to The OpenGov Foundation
– By Bill Hunt (@Krues8dr)
A few months ago, The OpenGov Foundation was awarded a grant from Hypothes.is’ Open Annotation Fund to add cross-browser support for Annotator, the tool powering inline annotation on government policy documents – like legislation and regulations – with Madison. For the civic technology community, this is no small obstacle: new, life-improving technologies often founder on the shoals of out-dated web browsers and operating systems. Some context: in the United States, for every person surfing the web with Chrome last month, two were not, according to StatCounter.
At the April 2014 I Annotate conference in San Francisco, we identified many older browsers, particularly Internet Explorer (IE), that simply could not run Annotator; therefore, large swaths of the world could not harness Madison. Since many citizens and government offices use older versions of Windows and Internet Explorer, in order to achieve our open government goals with Madison, we needed to modify the Annotator software package to support all of our users, not just those with new computers running the latest Internet browsers.
Creating an Assessment
The first task was to assess the scope of the problem. In today’s current web browser landscape, where does Annotator work as it should? Where does it break? Since Annotator has a built-in test suite, the fastest way to answers was to run the tests in every browser we wanted Annotator to work. To get a more comprehensive view, we needed to create an automated test runner and find a way to run our tests across multiple browsers.
For this, we used SauceLabs to create virtualized browsers to run our test suite against, since it contains almost every platform we care about. The SauceLabs API provided an easy way of creating these test runs, and the grunt-saucelabs package gave us an easy way to interface with them via Grunt. We investigated other solutions, but none were nearly as stable and mature. SauceLabs can also easily be integrated with most continuous integration tools, though we did not go that direction on this project.
Browser Testing Begins, Fails Immediately
While trying to run the initial tests in most versions of IE and older versions of Safari, the tests simply wouldn’t go. There were clearly major issues to investigate. Several of these problems turned out to be related to the way we ran the tests–some help from SauceLabs set us on the right track.
Next, we discovered that several of the testing libraries in use – including Chai and Sinon – didn’t offer out-of-the-box support for IE version 8. For Chai, we are only using the assert method for testing, and were able to make a straight replacement for assertive-chai. Implementing the bundled sinon-ie library solved the Sinon issue.
That was the easy part.
Patching for Features, Shim Libraries
The main problem with older browsers attempting to run Annotator is that they were written before current Internet standards became widely adopted, or in some cases, before those standards were even created. As a result, older browsers simply lack features from modern tools, or define and handle those features differently. In many cases, there are shim libraries available to make these features work properly, but not always.
In IE8, for example, the console api is only available if the Developer Tools window is open. When the window is closed (the default), using any of the console functions will cause an error. Since console sends the error messages in Annotator, it had to be replaced when not available. Since only a few lines of code are needed to replace console with an empty object, we wrote a custom shim to solve the problem. Similarly, there are a few calls to constants on the DOM Node constructor which is not present in IE. We therefore created a replacement Node object and hard coded those values.
Passing the Tests (Mostly)
With that done, most tests passed. However, Annotator still didn’t work perfectly in older browsers. A few features still eluded our tests, such as the handling of click events, which were inconsistent across browsers. We also had to do some tweaks to the CSS, as the rgba() css function, used for creating transparent colors, was not available. An opaque fallback color was used in those cases. The only other issue was CORS support, which we were able to manually enable in jQuery.
Wrapping Up & What’s Next
By the time we completed all of this work, we had run out of time for further improvements, like continuous integration and Selenium-based acceptance testing. And since the existing tests do not provide complete coverage, passing every test doesn’t guarantee that Annotator works perfectly in every browser, every time. While it will be up to the Annotator community to maintain and extend this project, we made significant progress. Thanks to the Open Annotation Fund and Hypothes.is, millions more Internet users can put Madison to work where they live, in whichever web browser they want – or have – to use.
Madison v1.7 Release Notes: All About Simplifying Complicated Policy Conversations for Citizens
By Chris Birk, Lead Madison Developer
The OpenGov Foundation is excited to announce the release of Madison Version 1.7 (v1.7). The latest edition of the Madison online collaborative policymaking platformis now available free for anyone to take and use on Github. Madison v1.7 includes several bug fixes and design enhancements, but it’s also loaded with a few new features to make it easier for you to engage in conversation with fellow users, stay updated on the latest Madison activity, and break down language barriers to bring in wider pools of expertise around policy issues. Madison is built for you, so please tell us what you think!
- Discussion Panel Now, general document comments are separated from annotations on the document text itself, allowing users to easily differentiate over-all citizen input on a policy document from actual edits to the document. The comments section functions more like a discussion forum than the previous activity feed. To join the conversation on a bill, click on the “Discussion” tab at the top of every document in Madison.
- Notification Engine Madison v1.7 now includes a notifications system. In this release, Madison Notifications are only available to administrative users. So if you are running a Madison instance for your city council or state government, for example, you will get instant updates on everything that is happening, from new document posts to new user signups and new activity on documents. The OpenGov Foundation’s developer team will be expanding Madison Notifications soon to give all users the ability to stay updated on the Madison action they want, like responses to their posts, new activity on a document they’ve previously interacted with, and even new versions of a document in which they’re interested.
- Social Logins Madison users can now log in and sign up with their Facebook, Twitter, or LinkedIn accounts. However, this addition has raised some interesting questions around the Madison signup process.
- Automatic Translations Madison v1.7 now includes the Google Translation Library to automatically translate document content into your language of choice. If Madison detects a browser set to a language other than the default English, you will be prompted to translate the page into another language if you’d like to do so. This new feature will help support the growing number of international Madison users.
- Document Activity RSS Feeds Each document in Madison v1.7 now includes an RSS feed, allowing you to track every document comment, discussion or annotation.
- Embed Madison Document Text on Other Websites Documents sponsors can now embed the text of any Madison document on any third-party website they want. It’s as simple as copying-and-pasting with the embeddable Madison widget.
What’s Next? Madison v1.8 is scheduled for release in late September 2014. V1.8 will focus on helping the Madison community stay informed and on top of the discussions and policy documents they want. There are plenty of other goodies in store with v1.8, like a step-by-step Madison how-to guide and custom policy document summaries from elected officials. Madison’s next milestone is coming fast. Watch it come together on Github, or sign up to be the first to hear about the new features. And we’re always Tweeting about Madison on @FoundOpenGov. We’d love to hear your feedback and suggestions for v1.8 and beyond!
Bonus Question for You on Social Media Log Ins: Before this release, Madison identified users and give them access to the conversation with email addresses only. Twitter, however, does not provide email addresses when using the service to log into Madison. The assumption that everyone would have an email address made it very simple for Madison to communicate with you. Now, we’ve had to rethink: should Madison ask you for an email address after you sign up with Twitter or move to a username-based login? Should we allow users to log in with either username or email, and if so, how do we set up notifications / communication for users who haven’t provided a contact email? Let us know what you think in the comments below!
August is upon us. We’re taking this quiet-ish time to review the year and recharge for the final sprint. Here’s how The OpenGov Foundation will remember 2014 (so far):
Launched the Free Law Founders — a nationwide coalition to modernize American lawmaking for the Internet Age
In July, we helped launch the Free Law Founders, a nationwide coalition of local elected officials, non-profit software developers, educators, and city attorneys dedicated to upgrading how citizens can access America’s laws, legislation, and the lawmaking process itself on the Internet. Learn more about what the coalition does and how you can become involved here.
Abhi Nemani, fresh from Code for America, joins OpenGov Board of Directors
In June we were honored to have Abhi Nemani, former Co-Director of Code for America, join our board. Abhi is a writer, speaker, organizer, and technologist who helped build the national non-profit, Code for America, from the ground up.
Citizens are reforming San Francisco laws with SanFranciscoDecoded.org
In July, the San Francisco Board of Supervisors voted into law one of the first ordinances created in the US through online citizen input. Supervisor Mark Farrell introduced the bill after receiving a comment on SanFranciscoCode.org, one of our America Decoded sites, highlighting a law that prohibits storing bicycles in San Francisco garages.
In a ground-breaking civic engagement initiative, Supervisor Farrell also launched ReImagineSF, a competition offering scholarships to students in the city who use SanFranciscoCode.org to suggest improvements to the city’s laws. Supervisor Farrell announced the winners on July 15, and will introduce their ideas as bills to the Board of Supervisors later this year.
“Decoding” city & state laws gets easier with OpenGov-AmLegal partnership
In May, we announced a partnership with American Legal Publishing Corporation that gives us easy access to well-formatted legal code from over 2,000 American Legal client cities across the nation. Through this partnership we have built a parser that allows us to more easily make American Legal cities’ laws accessible and restriction-free online. We’re exploring a similar partnership with legal publisher Municode, too. These partnerships enable us to scale our work in ways that previously proved difficult due to the lack of legal code in workable data formats. Click here to get started “decoding” your laws and legal codes.
Madison powering online citizen-government collaboration on DC legislation
We launched a beta version of our Madison collaborative drafting platform in the District of Columbia in partnership with Councilmember David Grosso back in May. In addition, Councilmember Grosso brought comments received on Madison directly into the hearing, using citizens’ suggestions and questions in conversation with public witnesses.
Our general Madison instance continues to host key pieces of legislation from around the nation, including seven bills related to open government and open data authored by New York City Council Member Ben Kallos, and Senators Leahy and Cornyn’s FOIA Improvement Act of 2014. And we have created a new instance of the platform at the request of the United Nations’s International Telecommunication Union, to support the crowdsourcing of a document concerning the use of information and communications technology in empowering youth.
Partnering with the W3C on open source legislative & legal drafting platform
We have partnered with the World Wide Web Consortium (W3C), which governs the architecture and rules of the Web, to create an entirely new tool that combines project workflow management, collaborative document editing, and intelligent information gathering. This tool will eventually replace the current editing tool on Madison, creating a powerful new way to approach policy-creation.
Won $7,500 from Hypothes.is to bring online annotation to older web browsers
In June, we were awarded a $7,500 grant from Hypothes.is to improve the open-source project Annotator, a tool for annotating the web, and which powers Madison’s annotations. The grant goes towards ensuring that Annotator works consistently across browsers and older versions of browsers.
“Design Whisperer” Jen Yu joins OpenGov Board of Advisors
We’re thrilled that Jen Yu has joined our Board of Advisors to serve as our “Design Whisperer.” She has led wildly successful design and experience teams, from Disney to Adobe, Slide (acquired by Google), Jawbone, and Frog Design, to name a few. On top of those amazing accomplishments, Jen has led the Defense Advanced Research Projects Agency’s (DARPA) experience design team in building Plan X, the U.S. Defense Department’s foundational cyberwarfare program. Right now, she’s the Chief Creative Officer of a Y Combinator startup in San Francisco. And she says that if you buy her a cookie, she will gladly take it.
That’s where we’ve been. Where are we going for the rest of 2014? Just ask!
Learning from DC’s First Legislative Crowdsourcing Initiative
“[T]he Madison project . . . is going to revolutionize the way we draft legislation and policy in the District of Columbia.”
– DC Councilmember David Grosso
In early May of 2014 we launched our first municipal instance of Madison along with DC Councilmember David Grosso, in what became the District’s premier legislative crowdsourcing initiative, and one of the first in the country. We also coordinated a live online committee hearing on one of the bills, Councilmember Grosso’s Urban Farming and Food Sustainability Act of 2014, on June 12.
In approaching this initiative, we had to consider how to measure success–how to determine if Madison was “working”. We concluded that anytime a citizen reads a policy document, watches a hearing, or interacts with policy who might otherwise not have had such access to their government, it counts as a win. But real success lies in whether public input on Madison reaches a document sponsor.
Of Councilmember Grosso’s documents, the urban farming bill received the most action on MadisonDC, with over 50 comments and 304 unique pageviews. The comments proved both thoughtful and substantive, and over 100 DC residents tuned in to the public hearing via the bill page on MadisonDC in the middle of a Thursday morning. But most importantly, Councilmember Grosso’s office reviewed all Madison activity on the bills, and during the public hearing, the councilmember used comments from MadisonDC to tease out finer points of the legislation in conversation with public witnesses. It formed a quietly groundbreaking moment in DC history, and our success with this document revealed some key lessons about using new technologies to facilitate civic engagement.
First, tools like Madison don’t have to completely disrupt the way government functions in order to be disruptive. Councilmember Grosso’s office was willing to use Madison because we collaborated with them to make sure Madison fit into their existing workflow as much as possible; if it had taken too much time out of their jam-packed schedules they wouldn’t have been able to use it. And the urban farming bill gained traction because community leaders activated their engaged networks to create interest around the bill. Leveraging existing resources remains key to creating real change.
Relatedly, it’s all about timing. We released the urban farming bill just as two council committees began actively reaching out to citizens for testimony. The committees held a joint in-person hearing on the legislation a month after our launch, giving us enough time to build interest and get people familiar with the concept of Madison, and providing an effective end-point to the initiative. It’s important to ensure that citizen input won’t just sit in Internet limbo, but also that people have enough time to get involved.
We also learned that yes, opening up the legislative process with new technology is worth the initial effort of getting started. The fact that Councilmember Grosso brought questions and suggestions received on MadisonDC directly into an official public hearing speaks to the usefulness of the conversation on Madison. And the fact that DC residents took the time to engage with the bill shows that they value the opportunity to become more easily involved in their government.
Finally, it’s tempting to think of revolutions as sudden and explosive, but in reality they constitute momentous amounts of small, incremental changes that lead to something new. The fact that our revolution lives primarily online doesn’t change that. Our platform is still in beta testing. So is the process of using technology to solicit citizen input directly on legislation. So we approached the initiative as an experiment, managing the expectations both of ourselves and our community and government partners. It was important for everyone to be on the same page and to keep an open mind if things went wrong.
Madison hasn’t yet changed the world. But small victories like the MadisonDC launch move us closer to a more inclusive and collaborative democracy, and it’s exciting to be a part of that change–one citizen comment and one line of code at a time.
Citizens, technologists and public officials working together to transform
state & local lawmaking for the 21st Century
Transforming America’s laws and legislation to meet the needs of the Internet Age requires a team effort. State and local government officials, residents and civic software developers all have a role to play. So do vendors, open government advocacy groups and cutting-edge research universities. That’s the only way we can “decode” the United States, opening up online the central data in every democracy: the laws and legal codes that govern us.
Enter the Free Law Founders (FLF). As Government Executive writes, the Free Law Founders is a nationwide “partnership to create new tools, data standards and processes for state and local governments to make public information and data better accessible to the public.” Why the FLF? The problems of today’s exclusionary, inefficient and paper-based laws and legislative process are faced by every single government in America. Go to any city council or state house in the United States – as The OpenGov Foundation has done over the last year and a half – and you’ll hear the same thing: our software stinks and we hate PDFs, but we don’t have the time, tools or talent to fix it.
That’s a problem crying out for cross-country collaboration and low-cost open source solutions. It’s “The Ultimate in Open Government.” To get there, FLF members will be working on expanding and enhancing free tools like Madison, America Decoded and Councilmatic. We’ll be working on crafting and passing policies to foster this open law ecosystem, with an open legal data standard to match. And we’ll be sharing everything – our code, our data and our gameplans – at FreeLawFounders.org so others can benefit.
As NYC Council Member Ben Kallos, a founding FLF member, put it: “The law must be free. The government must belong to the people, and with it the source code that operates and improves legislation and laws. Millenia ago, Hammurabi codified law and displayed it publicly for the people to see. Today, public means free and online, not behind a license or paywall.”
In addition to Kallos and The OpenGov Foundation, current FLF members include: San Francisco Supervisor Mark Farrell, NYC Council Member Ben Kallos, Washington, D.C. Council Chief Counsel David Zvenyach, Chicago City Clerk Susana Mendoza, Boston’s Principal Data Scientist Curt Savoie, the Participatory Politics Foundation, and MIT’s Human Dynamics Lab. As FLF members, we’re sharing our time and talents to vivify these basic principles of Free Law, and contribute to securing them for the entire United States.
The Free Law Founders is open for anyone to join. Civic hackers, everyday Americans, government officials, private-sector vendors – everyone can contribute to achieving this massive modernization effort of how we make and access America’s state and city laws, legal codes and legislation. Drop me a line to get started on the path to transforming your city and state laws and legislative process for the Internet Age.
Don’t get stuck in the Dark Ages of Democracy with PDFs, copyright restrictions and chaos. Do something about it with the Free Law Founders.