UnScrivening

I truly think, I just fell in love with a program. I’m talking about Scrivener. Since I’m doing research / academic writing rather than novel writing, I had not really looked at the program after I had bought it and played a bit with it a couple of years ago. Now, I’m starting here a series of blog articles about how to make Scrivener really work for academic writing and research. It’ll go very much in detail.

Motivation

First of all, here’s my motivation: I used the obvious combination of Sublime Text, BibDesk and LaTeX, for writing about 120 papers during my MBA. I even published, at some point, my template to GitHub. Don’t really download it, as I’ve changed it over the last year, and I am now substantially redoing it.

But anyway, that template allowed me to have a standard layout and template for all my papers – every week I would clone it, write whatever was there to write, run through the Makefile that’s embedded there, and done.

Yet, the more I think about that, having an approach where I have essentially my layout of a fixed set of chapters, works conceptually only if you mostly already know what structure you are going to follow. LaTeX is very good, of course, at re-arranging stuff and keeping all your references intact, but then you’d still reshuffle content between chapters.

That works well for an MBA, where people will tell you, read those articles, write something about that question about them. I mean, I honestly believe that the pure visual excellence of my papers was a big part in my becoming Student of the Year at Liverpool university…

“Your paper makes no goddamn sense, but it’s the most beautiful thing I have ever laid eyes on.” Source: Internet (Peer reviewed, trust-worthy).

Now, what changes when you go from MBA, paper writing mode, to DBA/PhD, research mode? Essentially, from a research standpoint, you probably have no idea what to read, so you’re doing an actual literature research. You have bits and pieces of information flying around.

In other words, you don’t have your couple of chapters that you know in advance. You’ve fragments that you re-arrange all the time.

Enter Scrivener

Now, Scrivener very much can be seen as a management interface for text snippets. So rather than having chapters, you may have bits and pieces of information:

Scrivener – Bits and Pieces all over the Place

Forget about these funny ### for the moment. I’m going to write a different post on that. I’ve written TeXDown over the last couple of days, and if you don’t want to test it right now, then wait for the blog post that I’m going to write about it soon. But besides parsing out ### and replacing it with something that LaTeX understands (like, \section{…}\label{…}), what I found myself very quickly developing TeXDown into was much more than a parser for that markup (actually, MarkDown) code.

Because, in order to parse the content that was in Scrivener, I needed to parse the Scrivener data structures, and also its directory hierarchy.

Scrivener Data Structure

So here’s what I’ve learned in the meanwhile. And I don’t claim I understand everything (for sure, I can only speak about Scrivener on a Mac), but here’s what essentially happens: A “Scrivener” file, let’s name it “Dissertation”, is actually a directory “Dissertation.scriv”, with a whole lot of files in it.

Let’s look at the details:

1. Every single node in the above mentioned screenshot of Scrivener, if it has some text in it, will have a corresponding file on the disk, in a subdirectory Files/Docs.
2. Every single node has a unique, numerical, ID, let’s say, 123.
3. The file in Files/Docs is named, then, 123.rtf.
4. At the top level of the Dissertation.scriv directory, there’s one XML file, named Dissertation.scrivx. That file contains the mappings between the content that you see, i.e., your hierarchical structure, and the actual files on disk.
5. There are also some more files related to 123, so for example,
   1. Snapshots/123/*.rtf which timestamps as filenames (see screenshot above), as well as each one file Snapshots/123/index.xml that contains the mappings of snapshot files to names of snapshots that you can potentially give.
   2. Likewise, there’s things like Files/Docs/123.comments, which exists when you use footnotes or comments in scrivener on your document fragment, and these files are xml files that contain, for each footnote, an rtf code in a CDATA field; the 123.rtf, then, contains a hyperlink with a unique ID for each comment.
   3. Ultimately, if you also have written something on the front card (synopsis) of an asset in Scrivener, you’d have e.g., a file Files/Docs/123_synopsis.txt.
6. You can, in your Scrivener frontend, actually have conflicting names of assets: It is no problem to do something like a folder /x, and having two assets y in it, with different content. That works, because essentially, the file content is mapped via indirection by the .scrivx file to what you see.

Enter Git

Git is of course the most awesome versioning tool on the planet. Now how can we make it work with Scrivener? If we just throw something at Git – like, the Scrivener directory, and – by virtue of .gitignore, exclude the stuff that changes too frequently, and is too irrelevant for us…

*/Files/binder.autosave
*/Files/binder.backup
*/Files/search.indexes
*/Files/user.lock
*/Files/Docs/docs.checksum
*/QuickLook/
*/Settings/ui.plist

we can do this in the directory where we have our Scrivener “file” (like, where we have Scrivener.scriv):

rm -rf .git
git init
git add .
git commit -a -m "Initial version"

This will create a new Git repository locally for us, add everything, and then commit it. Once we’ve done that, let’s fire up the most awesome Git front end on the planet, Tower (it costs about 80 bucks, but really, do get it…), and put our content into a branch A:

Scrivener seen from Tower

So on the right side, you can see the effect of the fact that from a file system point of view, you have only “numbers” for asset names, and also you have them all on the same directory level.

And, you can see that in the preview, you see some text, but you also see a whole lot of rtf markup, because, well those are rtf files after all.

While Git has no problem with either – rtf files are text files – from a usability standpoint, both issues are not quite easy to solve. Let’s look at an effect of it.

Let’s edit the file in Scrivener, change something, like add some text and a footnote:

Using Scrivener to amend some file

Here is what Tower now sees, when we close Scrivener, and look at Tower:

How Tower sees differences in Scrivener files

On the right side, in green, you see the changes that Tower has seen, vs. the previous content, which is shown in red. Now, if link Tower up with the most awesome diff tool on the face of this earth, Kaleidoscope (it costs another 70 bucks, but really, do get it…), we can compare both versions and see the changes in a much clearer way:

Kaleidoscope is good at comparing RTF…

Kaleidoscope doesn’t show us images, should we have embedded them in an rtf – which I can live with, since I would probably not integrate an image into the rtf at this point – I’d rather use the LaTeX way of linking to the image on disk. But maybe I’m going to change that – after all, I could use Scrivener for that too, and when running TeXDown, I could extract those files, put them into a directory for LaTeX, then add around the place where they were a template for images that I anyway use all over the place, and then this way have a much easier way of adding images to my stuff. I don’t have so many images anyway, but that might be an option. And since tables suck both with MultiMarkDown – and hence I was of course to lazy to even put anything about tables into TeXDown – as well as with LaTeX, I could just create the tables some other place, put them into PDF, and then use the mentioned approach to have Scrivener have a placeholder for me, also have the PDF, and then have TeXDown take care of all the rest. But I’m digressing…

Anyway, let’s create the current version into a branch b. Then merge b back into a, which will of course create a nice merge conflict.

Tower sees a merge conflict

Which we then can look at in Kaleidoscope:

…but Kaleidoscope cannot merge RTF

Notice those buttons in the status bar, right in the middle? If it was a text file, those would not be greyed out, but you could choose, for each merge conflict, whatever side you prefer, into the merged content in the middle.

Just, it doesn’t work with rtf, which leaves us with two problems:

1. How can we make better sense of those 123.rtf filenames, all living in one directory, seen from the point of view of the file system / Git / Tower?
2. How can we actually make Kaleidoscope able to merge the content?

Possible Solutions

File System Level

What we’ve seen so far, is that the file system level operations against a Scrivener project are possible, but also not very intuitive – even for an avid file system / shell user, simply because you’ll have to link whatever 123.rtf is to whatever you’ll see in Scrivener.

My little tool TeXDown, that I had linked to above, can help us here: It actually allows you to get that mapping:

$ texdown.pl Dissertation -l -p / | grep 123
[     123] /Research/ROI/Material/Discussion with Clayton Christensen/Daten/Breite der Daten, Ergebnisse, Hypothesen

But then, we’d logically want to re-create the hierarchy that you have in Scrivener, on the file system. Hence, we would have to expect the users to not do either of two things:

1. Have naming conflicts on a given hierarchy level
2. Have characters in the object names that won’t work on the file system (like, a slash)

Both we could of course deal with, but then we’d have to somehow be clever about the mapping. For example, we could remove all those incompatible characters, replace them by spaces, and also add the “123”, i.e., the object ID, to the file / directory name. We could then both ways export and import the files using some little script.

Summary: File system level: can be solved. We would create a folder hierarchy that is disjunct from that Scrivener works on, and which we can then work on with Git. Essentially, the workflow would be like this:

1. Exit Scrivener
2. Run script to dump Scrivener content into a file system hierarchy
3. Have Git look at that content
4. Do something, like branching, merging
5. Run script to re-import Scrivener into Scrivener’s directory
6. Start Scrivener

This can clearly work, except for the fourth point. Let’s look at that.

File Level

Yes we can export Scrivener stuff into our own hierarchy. But what can we then see? As we saw before, visually, we would now not see like 123.rtf all in one directory, we would see some actual names – but still:

1. The file content would appear mostly garbled
2. We can diff it, but not merge it

So to solve that, there are several options, and neither is perfect in anyway. All of them come at some cost. All of them involve converting rtf into plain text, and back – which by definition comes with a loss.

Convert RTF to Plain Text

MacOS comes with a nifty tool called Textutil, which is able to convert RTF into plain text:

Using Textutil to convert RTF to TXT

As you can see, that basically works, but it of course looses all other information – like, our footnotes. And really, there’s just also other ways to convert RTF to TXT, like Gnu’s unrtf, or like this little snippet of Perl:

#!/usr/bin/perl                # <= Adapt if needed
###################################################
#
# rtf2txt.pl
# 
# Convert a RTF ot Plain Text
# 
###################################################
# (c) Matthias Nott, SAP. Licensed under WTFPL.
###################################################

use strict;
use warnings;
 
binmode STDOUT, ":utf8";
use utf8;
use RTF::TEXT::Converter;

print rtf2txt($ARGV[0])."\n";

#
# rtf2txt
# 
# Convert a file from rtf to a txt string.
# 
# $file     : The file to convert
# 
sub rtf2txt {
	my $file = shift;
	my $result;
	my $self = new RTF::TEXT::Converter(output => \$result);
	$self->parse_stream($file);
	return $result;
}

I know, it is not nice, but now let’s throw some rtf at it:

$ rtf2txt.pl 123.rtf 
HelveticaNeue;###[Coakes, Smith, and Alwis (2011)] [t#Coakes:2011aa]

% Coakes, E. W., Smith, P. A. C., & Alwis, D. (2011). `Sustainable Innovation and Right to Market', Information Systems Management, 28 (1), pp. 30-42. [Online]. Available from: http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=57303238& site=eds-live (Accessed: November 6, 2016).

Create Version a here. Oh yes, with a comment, and a footnote. => and yes, here is some more

content. And one more footnote.

See that second line? For some reason that sometimes happens. Just running it again may, or not, show that HelveticaNeue;…

$ rtf2txt.pl 123.rtf 
###[Coakes, Smith, and Alwis (2011)] [t#Coakes:2011aa]

This seems to be some problem with the RTF::TEXT::Converter module. I’ve seen that with other people’s code, too.

Anyway, we could still live with that, but do we want to live with losing our footnotes? I guess not. Well then, we can also use Textutil to convert into HTML rather than into plain text:

$ textutil -convert html 123.rtf 
$ cat 123.html 



  
  
  
  
  
  


###[Coakes, Smith, and Alwis (2011)] [t#Coakes:2011aa]


% Coakes, E. W., Smith, P. A. C., & Alwis, D. (2011). ‘Sustainable Innovation and Right to Market’, Information Systems Management, 28 (1), pp. 30–42. [Online]. Available from: http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=57303238& site=eds-live (Accessed: November 6, 2016).


Create Version a here. Oh yes, with a comment, and a footnote. => and yes, here is some more


content. And one more footnote.

 

As you can see, it does maintain our footnotes, and I’ve tested that we can convert back and forth between html and rtf, and it will still work with Scrivener (always provided Scrivener is closed during external file manipulations).

But then again, this replaces one hard to read format (rtf) by another hard to read format (html) – it adds so much more stuff to the file that the actual textual content is even harder to see. It allows us to formally merge, with Kaleidoscope, but watch how that now looks:

Kaleidoscope doesn’t work so well with HTML – how could it?

So while we can now formally merge, it really doesn’t  help so much, as the real content is much harder to read, from Kaleidoscope or any other diff tool, compared to what we had before.

So what, do you think, could we do about this? We could of course solve this problem by yet another indirection, e.g., by stripping out all html content and marking up the remaining textual content with some IDs so that we would still remember, which bits of text we had were. For example, by wrapping every text into something like |id|text| and remembering some place, where that text lived in the HTML, we would have an option to be able to merge text, and then re-merge back in the html – and convert that back into rtf using Textutil. We’d still have to deal with images, which Textutil extracts into bmp files, but let’s assume we can somehow do that.

Conclusions

For me, while I really started to love Scrivener, this simple thing is something that makes me not like it – and yet I also must say, I don’t have so much of a better option really. At the end of the day, conceptually of course you end up getting dynamically named, e.g., numbered file names if you let the user shuffle them around. And why would you not allow for naming conflicts, because what you don’t want to do is to nag the user when he drags some content into some folder where there is already something by the same name. And why would you not use RTF, because after all, it allows the average user to do some formatting, and some images.

That leaves me with these options:

1. I can well make use of Git. Git understands of course RTF, and
2. Kaleidoscope will even nicely show the RTF in comparisons – we just won’t be able to merge there. We’d have to do it in Scrivener (and somehow as well, get around that while in a merge process, we’d be having files on the disks modified by Git which we are not able to make Scrivener work with).
3. I can also extract the whole Scrivener content into a directory tree – even two ways.
4. Scrivener is not able to version whole projects, it only does it on a per node level. We could potentially even hack this in by explicitly creating versions using a script – and Scrivener would pick them up.

This last point is something that is most interesting for me – at the end of the day, the whole thing is about versioning, which Scrivener does only half way – their programmers should really watch this video. They should really consider supporting a real versioning system from their point of view, which is simple right out of the box – and if they would do it, they could take care of the pre- and post processing and hence enable Git functionality on their rtf content.

What do you think?