free-style.text

https://twitter.com/scribblingfox/status/851289517872922624


I'm going to speak with your today about the verifibility crisis in
science, tech writing and life. 

So what does that mean?

I'm going to suggest that in recent years, we have experienced
issues with our ability to verify information in each of these domains.

From a tactical standpoint, this talk is addressed primarily toward
writers who write for a developer audience. But from philosophical
standpoint, what I discuss on our social experience should be relevant
to most of you.

Our story begins in XXXX with The Experiment Experiment. This was a
project initiated by XXXX in which they identified 100 published
peer-reviewed research papers. For each paper, a team of investigators
studied the methodology described in the paper and then attempted to
recreate the results that the authors of the paper obtained.

What the investigators found is that they were able to replicate the
original research from only 29 out of 100 papers. So 71% of the research
could not be replicated.

This experiment go some press and other people did similar
investigations and got similar results.

Here are some numbers from Wikipedia.

Now, before we jump to conclusions, we don't know precisely why the
results of these studies couldn't be duplicated. It could be that the
particular conditions of the original study couldn't be recreated, or
maybe the paper didn't provide enough information to enable the the
experiment to be duplicated, or perhaps the paper misrepresented the
results of the study.  We really don't know, but irrespective of the
reasons, this inability to replicate--or verify--the published results
of scientific research potentially creates a lack of confidence.

At this point, I want to introduce a word: veracity. One of the great
things about speaking at a technical writing conference is that I don't
generally have to be concerned that my audience won't understand a given
word. That said, here is the definition.

DEFN: Epistemology

DEFN: Veracity

So veracity means can some person--or information--be trusted in its
correctness, accuracy, truth.

I'm proposing that verifiability ensures veracity. 

[ logic diagram with implies ]

A fine point here, verifiability doesn't mean that you actually have to
do the work of verifying the information, it means that you _could_. 

Back to our story...

After The Experiment Experiment, other investigators conducted similar
studies with simiar results and all of this led to what became know as
the Reproducibility Crisis in Science.

Research scientists feeling assailed by all of this were looking for a
way to respond.

So in response, Roger Peng, a professor in the department of
biostatistics at Johns-Hopkins developed a set of guidelines for
Reproducible Research.

Reproducible research doesn't require that a study be replicable. Peng
felt that, as a standard, that was too high a bar. Instead Reproducible
Research says that a published paper must also make available the raw
data that was collected and the code that was used to analyze the data.

Enter: RMarkdown

This guy Yihui blah blah saw that an elegant way to support reproducible
research would be to extend the Markdown language to enable R
programming code to be embedded in Markdown documents.

What is Markdown?

Markdown is a light-weight markup language that has received wide-spread
adoption in recent years. It provides and simpler authoring experience
than an industrial strength XML authoring environment. It was invented
by John Gruber of Daring Fireball. 

Provide links to gruber and DF twitter feeds.

What is R?

R is an open-source programming language that is designed specifically
for statistical analysis.

Here is a Markdown document. 

Here is some R program code that calculates the Fibonacci sequence.

And here is some R code embedded in a Markdown document. 

And here is what that looks like:

So by default, the output from R is included in the document text.

When you build the RMarkdown document, the embedded code is executed and
at your option, the output is included in the published document.

The implications of this for researchers is that the electronic version
of the research paper doesn't just display the code that was used to 
analyze the data, building the document actually recreates the analysis
of the data.

This way, anyone reading the paper has an assurance that they _could_
reproduce the results that they are reading about.

The operative word here is _could_, but what if they actually wanted to?
The published document, for simplicity, doesn't show all the source code
and doesn't contain all the data. 

---

Knitr Output

RMarkdown leverages Pandoc to process the markdown component of the
source document. So RMarkdown supports any output format supported by
Pandoc.

And I have kind of a funny story about that . . . at HBO, the technical
writing team is part of the larger Design organization and I report to
the VP of design, Ryan. Well, Ryan is a Keynote afficianado and was
somewhat aghast at my slides . . . I am not a designer.


[Reproducible workflow]

This is where GitHub comes in. 

(This is going to get a little geeky, but please bear with me.)

The RMarkdown source for the document is stored on GitHub along with the
data that supports the research. The document links to the GitHub
repository that contains the source and data.

[Make]
GitHub solves the problem of making the code and data available. But
what about actually reproducing the research--which is to say recreating
the document?

The standard tool is Make, which is a tool from Unix that can encapsulate
all the steps necessary to build the document. 

So the reproducible workflow reduces to:

Copy the GitHub repository to your local computer (`git pull`).
Install R (e.g. `brew install R`)
Make

In practice though, most readers won't need to actually build the
document, the mere fact that they could (and that the authors had to in
order to publish) creates verifiability.


Reproducible Research and RMarkdown has taken hold in the Data Science
and Statistics communities, but it hasn't been all Rainbows and
Unicorns. In <MONTH>, the New England Journal of Medicine published an
op-ed that said that they were not getting onboard with Reproducible
Research. Their view is that the data that comes out of scientific
research is the IP of the researchers and papers that our published in
the New England Journal will not be accompanied by data. 

So we will have to see how this all unfolds over the next few years.


Okay, so we've briefed ourselves on the reproducibility crisis and the
methods that the data science and statistics communities have devised to
counter it. Now, I want to talk about how this effects our field,
technical writing, as well as broader information environment.

First technical writing.

As someone who has written a lot of developer-facing documentation, I
often find myself thinking about Stack Overflow--and what makes Stack
Overflow so compelling for so many developers.

I think there is more than one reason, but I want to focus on one in
particular, and I want to illustrate that by telling your about an
experience I had when I first go out of college.

My first job was doing technical support for Intel Corporation not far
from here actually in Hillsboro, OR. This was 1988 and we did not
have email, just to give you a sense of what we were dealing with at the
time. My job was to help Intel's engineering customers use the 386
processor. The 386 could operate in different modes _Real Mode_ and
_Protect Mode_. One of my customers was stuck trying to transition the
386 from real mode into protected mode. So, I sent the customer some
sample code that performed that operation. 

About a week later, my boss received a letter from the customer. A
letter, because as I mentioned: no email. The letter expressed how happy
the customer was with the service they had received. In particular, the
customer was astonished that the sample code that I had had actually
worked. Evidently, this was not typical. This was 1988.

A lot has changed in our industry since then, but this issue of sample
code not working is perrenial. Every doc team seems to have their own
unique strategy for dealing with it. I myself have participated in
designing some of the most byzantine systems for extracting code
snippets from source files and then injecting them into the
documentation. 

You can probably see where I am going with this. Technologies _such as_
Knitr are the best thing we have to a solution to this problem.

The reason is that Knitr enables you to keep the documentation and the
source code in the same file. Most other solutions involve with
extracting documentation from the source code or extracting code
snippets from the source code and injecting it into the docs. 

Also, everytime that the document is built and published, the embedded
source code is executed, which is to say it is tested. So you can't
publish the document without testing the embedded source code.


Language Support

When Yuhui, designed knitr, he foresaw that researchers might want to
use programming languages other than R for their analysis. So it made
the language processor extensible . . . and then proceeded to extend it.

Here is a list of the languages currently supported in RMarkdown. So,
here is fibonnaci written in Python.

[Code Snippets]

Knitr also solves the issue of keeping the code samples concise.
Often times, you don't want to expose, in your document, all the
ancillary code this needed to call a function or method--but on the
other hand, without that code you can't actually verify that you are
calling the function function correctly.

Knitr solves this issues by enabling you to make visible only those
parts of the code that you want to focus on. The rest of the code is not
shown in the published document, but is still executed. 

The reader can view the ancillary code by inspecting the source for the
Knitr document on GitHub.

Now, so far, I have been focusing on Knitr, but really Knitr is just one
implementations of this idea. Other examples are:

- Bootstrap documentation
- Jupyter notebooks
- Storyteller 

Bootstrap is a technology invented by Twitter, but now opensource,
which makes it easy* to add sophisticated styling to HTML pages.
Because the documentation for Bootstrap is published in HTML, the
documentation itself is able to demonstrate the Bootstrap functionality
being demonstrated.

[Example]

Jupyter notebooks are a technology from the Python world which enables
you to combine expository text and code in a single interactive document. 
Jupyter now supports multiple languages.

Storyteller is a technology from Microsoft that enables the creation
of _executable specifications_. Storyteller solves a different problem
that Knitr or Jupyter, but it derives from the same idea of an an
executable document.


[Verifiability and Fake News]

Twitter: Urban Dictionary: Fake News
<https://twitter.com/urbandictionary/status/850804700903612417>

In the modern world, misinformation has become high art, an expression
of irony.

Or to take a less optimistic view, the art of misinformation has become
the art of war.

And sadly, the issues with reproducibility in the scientific community
have contributed to the problem. 

But I'd like to suggest that the same tools that have been invented to
counter the reproducibility crisis can--and eventually will be--brought
to bear on the fake news crisis.


I'm sure that many of you are aware that Hillary Clinton won a majority
of the popular vote. And obviously, President Trump won a majority of
the eletoral votes.

But there hasn't been a lot of discussion about how the popular vote
margin, affected the electoral vote margin.

Well, Nate Silver published an article titled 

What a difference two percentage points makes
https://fivethirtyeight.com/features/what-a-difference-2-percentage-points-makes/

So let's look at an intelligent document that verifies that what Nate
says is true.

---

Fake news often takes the form of statistical information. 
You see a lot of statistics sloshing in the media. I see that stuff and
I think what I earth?

[Kirk]

I think: how am I supposed to interpret this stuff.

Oh, that statistic over there. That sounds pretty good. I'll believe
that. But hmmm, that statistic over there. That doesn't sound so good. I
won't believe that one.

These methods that come to use from Reproducible research. Ultimately,
these methods are the solution to that problem. Providing a reference is
good. But in the future, the gold standard will be embedded verifiability.

I want to see a future, where every statistic provides either the code
and the data or a reference to a reproducible research paper that
provides the code and data.

And then, we can put an end to fake news for good. Thank you.


--- END ---