Installing Calibre and Plugins

(My site has decided to eat all the images I posted here, which was causing endless load loops.  To stop that, I took them off.  I don’t know why it does this.)

Here’s a detailed and hopefully helpful walkthrough for using Calibre with AO3, using the FanFicFare plugin.  Follow these steps to download fic, keep track of what you’re reading (or not reading), and sync eBooks from one device to another.

The first thing you need to do is download Calibre.  It currently has releases for Windows, Mac, and Linux.  I’m running Windows and iOS, which you’ll notice isn’t officially supported.  I’ll show you how to make that happen in a bit.  There are a few ways to do it.

Installing Calibre works like anything else.  Download it, run the file you’ve downloaded (whichever format it happens to be), and on we go to the next step.  When you run the Welcome Wizard, make sure your library is in a cloud folder.  Dropbox is best.  This will make syncing to a mobile device easiest, because the baked-in methods have got unnecessarily complicated.  If you don’t have Dropbox, that’s fine.  There are other ways to do it.

Once installed, the first thing we want to do is install FanFicFare.  For the sake of my own sanity and not going back through everything I’ve already done, I am not doing this from a clean install, so just pretend that my screenshots are blank.  Click on the Preferences button on the top of the menu, and then click on plugins on the bottom of the pop up.

Then, click on the Get new plugins button on the bottom, with the yellow jigsaw piece.

And then in the filter by name box, type in fanfic.  The only one that pops up at this point should be FanFicFare.  Select it and click install.

It may give you a security warning about not being monitored or supported or something, and ask you if you accept the risk.  Click OK or Accept or whatever affirmative button pops up (I’m not sure exactly and don’t want to reinstall this whole thing).  Optional, you can do this again for the Generate Cover plugin, if you want to make some fancy covers for your fic.  Or you can use the default ones Calibre gives you, which aren’t half bad themselves.  I just got bored.

Configuring Calibre for Use

Now, we need to configure FanFicFare.  You will need to give it your AO3 credentials in order for it to see locked and explicit fic.  Click on the down arrow on the side of the FanFicFare icon and select Configure FanFicFare.  Then, click on the personal.ini tab.  Scroll until you find the Archive Of Our Own section, and make sure it looks like this, adding your username and password:

[archiveofourown.org]
[defaults]
is_adult:true
username:XXXXXXXX
password:XXXXXXXX

If it’s not there, add the above text to the bottom of the document and replace your username and password with your own.  Calibre and FanFicFare cannot make changes to your account.  You are giving it read only access to be able to see the site and scrape fic content that is otherwise hidden from view.  If you are concerned about safety, you can create a separate account only for Calibre to use, but it’s not really worth doing since it would mean waiting in the invite queue.

There is more customisation to be done within the FanFicFare plugin, but it’s easier to do after you customise Calibre itself at this point.  If you right click on any column header, you’ll get a context menu with an option to add your own columns.

This is where you’ll be able to set your own custom information, such as read status, word count, ships, characters, and more.  Some of them can be auto-populated from AO3’s metadata, others will need to be manually entered.  There are loads of options, and it’s easy to use the wrong one in the wrong situation.  Rather than walk through every option a person could possibly want, I’ll explain what the options do and how they might be used in this context.

  • Lookup name: This is how Calibre stores the column information.  It should be all one word, and in lowercase.  If you do it wrong, it will tell you, and give you a pop-up telling you how to fix it.
  • Column Heading: This is how it looks client-side.  It can have capital letters and spaces and look nice.  If the lookup name is readstatus, your column heading can be Read Status.
  • Column Type: This is the tricky one that can cause a lot of confusion, because there are a lot of options.  I haven’t even used all of them yet, but here are the ones you may find the most helpful:
    • Text, column shown in the Tag browser: The default.  You get one entry and it doesn’t matter how it’s formatted.  Anything that goes in this field is the whole entire tag, even if it contains a comma.
    • Comma separated text, like tags, shown in the Tag browser: Proper tags.  Multiple entries, separated by commas, how you’re probably used to tags working.
    • Long text, like comments, not shown in the Tag browser: A space for notes to yourself.  Leave yourself reminders about where you’ve left off on a fic, or keep records of your own comments, or put anything you like in here.
    • Text, but with a fixed set of permitted values: If you want a drop down menu, and know you’re not going to be adding any more options, you can use this instead of the default text option.
    • Date: What it says on the tin.
    • Integers: Numbers.  Calibre alphabetises numbers.  I mean, three comes before two, because of how they’re spelt.  If you want to include word count or chapters, you need to use integers.  Ask me how I found that out.  Has options for formatting numbers, but you need to get into Python for that, and you probably won’t need to get that detailed.
    • Ratings, shown with stars: What it says on the tin.  Has an option to allow half-stars
    • Yes/No: What it says on the tin.  Has a few display options.  You can use this for whether or not you’ve read a fic, but I choose to use the default text so I can mark fics in progress.

There are a few others that I haven’t touched on, because they are a bit too fiddly and not really.  The text column for series-like information, and columns built from other columns are something you can play around with if you like, but I haven’t personally found them terribly useful.  All of these will have a field for Default value, which you can set to save yourself some time.  For instance, on a Read Status column, you can set the default value to Unread, which will save you having to manually set the status for every new import.

This is useful not only for tracking read status, but loads of other AO3 metadata as well.  By default, Calibre will import all of the basic tags, such as fandom, characters, ship, etc, but it will lump them all together into the “tags” category and not separate them out very well.  You can use custom columns to separate that information out, so that characters, ships, ratings, etc are easier to filter.  So far, the only one I have not been able to get to work is fandoms, which I have been doing manually.  You can also add in columns for word count and chapters, which is where those integers come in handy.  Another one I’ve found useful is to replace the default Date column, which only has the month and year, with a Date Updated column, which includes the day the fic was posted or updated.  You can’t delete default columns, but you can hide them from view, which is more or less the same thing.

Once you’ve got your custom columns set up for the information you’d like to track, Calibre will ask you to restart.  This is necessary for the next step, so do that.

Then, you’ll go back to configure FanFicFare, and click on the Custom Columns tab there.  Here, you can have FanFicFare pull any metadata from AO3 and match it up to the columns you’ve created.  It’s likely not everything you created will have a matching option, and that’s fine.  You can do some things manually, and some things by default.

Next, go to the Basic tag on the FanFicFare configure menu, and take a look at how you’d like to set up your reading and import experience.  I’m not going to tell you how to set this page up, because it’s going to vary depending on how you like things done, but I can explain some of the options.  I won’t go through all of them, because it’s a lot, but here are some of the important ones:

  • Default Output Format: When you scrape AO3 for fic, this is the format the fic will download as.  You have the options for ePub, MOBI, HTML, or txt.
  • Default of Story Already Exists: In this one case, I will say set it to Overwrite Always.  Otherwise, you won’t catch new chapters or edits when they’re made.
  • Keep existing tags when updating metadata?: If the author on AO3 deletes a tag, do you want Calibre to delete it as well?
  • Check for existing series anthology books?: Do you want Calibre to store series information?
  • Check for changed story url?: This will check if a oneshot has turned into a multi-chapter fic.
  • Take URLs from clipboard?: This saves you the step of pasting the URL from AO3 into a tiny little box.
  • Reject without confirmation?: If a fic is on your reject list, do you want Calibre to skip the fic entirely?
  • Delete on reject by default?: If you reject a fic, do you want Calibre to delete whatever information it has already stored?

There’s quite a lot more here, and I strongly encourage you to play around with all of them.  You can’t break anything, and if you don’t like the way something imports, delete it from your library, change some settings, and try again until it works the way you want it to work.  In fact, you’ll probably notice that we’ve ignored Calibre’s settings entirely, and only played with FanFicFare’s.  Calibre’s settings and preferences are much more intense and far less user friendly, so we won’t be touching them as much.

If you’d like to play around with the Generate Cover plugin, it’s a fairly intuitive plugin to use.  Again, you can’t break anything.  Just play around with it and see what happens.  Since it’s a visual plugin, it’s a lot harder to explain without screenshots, but if you need help I can show you in the discord.  Import some images, create some templates, and apply them to different fics.  I’m working on figuring out if there’s a way to apply covers to fics based on specific metadata, but so far it seems like that’s not possible.  I’ll update that page if anything changes.

Importing Fic

Now here’s the bit you’ve been wanting to get to.  And there are multiple ways to do it.  The way I did it is stupid and it will take all day, and I will show you how to do it that way if you like.  Or there’s a far more sensible way of going about it.  It amounts to more or less the same thing.  We’ll do the simple version first.

Importing fic one at a time

To import a single fic, copy the url from AO3 and click on the down arrow on the FanFicFare button on the menu, and then select the Download from URLs option.  Depending on your settings, the URL may already paste into the big box.  Double check that the settings are as they ought to be, and then click OK.  It will take its time fetching metadata, which for a single fic, is usually just a few moments.  Then, you’ll get a pop-up in the lower right corner, telling you that it’s complete.  If it’s worked, it will tell you that it found 1 good and 0 bad updates.  Click the Yes button to add the fic to your library.

Importing multiple fics

If you want to import an entire fandom, or maybe there have been a few posted at once that you want to read, you can do multiple fics at once.  And there are a few ways to do it.  You can do it the same way as above, pasting URLs individually, one per line, and then follow the same process.  This is best if you only want a few fics.  If you want to download an entire fandom, or you already have done and you want to update the day’s uploads, you can do a bulk download.

Go to the page in question that lists the fic.  This is going to be any page.  Say, for instance, this page.  Copy the URL, click that down arrow on the FanFicFare button, and this time select Get Story URLs From Web Page.  Make sure your URL is pasted in correctly, and then click the For Individual Books button.  It’ll take a minute for something to happen, and you may think that the whole thing has frozen.  Don’t worry about it.  Eventually, that same dialogue box as before will pop up with 20 URLs listed from that page you just pasted in.  Again, make sure everything is as it should be, and click OK.  It’ll take a lot longer this time to make import everything, but it will get to the same end result, with the entire page of fics importing.  I find on average, it takes about 5-6 minutes per page, but this can skew longer if there are more longfics than usual on a given page.

Updating fics

There are multiple ways to update fics.  When you import multiple fics using the method above, if there are any fics already in your library, Calibre will update them if they need updating.  When I go in to grab the 2-3 new fics that may be uploaded on a given day, it will also catch the ongoing WIPs that have also been updated as well.

You can also update fics by highlighting them in your library and just clicking on the FanFicFare icon directly.  Whether you do it one at a time, or in a batch, you’ll get the same window, titled Update Existing List.  It will have similar options to the window above.  Check those to make sure they are set to do what you want them to do, and click OK.  Calibre will then go through the same process as before, to fetch the metadata from AO3, and then download the fic from the site.  I generally find it easier to do it the copy/paste way, because in almost every situation, updated fic will be moved to the top of the tag anyway.

Editing Metadata

The only time you’ll want to really use Calibre’s built-in options is when editing metadata.  You can do this in bulk or individually, and the times you’d want to do either depend on which metadata you’re editing.  In most cases, you really need to only double click on the relevant field right there in your library and it will let you edit it, but sometimes that will pull up the book because it thinks you want to read it.  You can right click to get a context menu, which will have options to edit metadata individually or in bulk, or use the metadata icon in the menu to do the same.  To edit metadata in bulk, you will need to have multiple items selected, and to do that, click on one item, hold down shift, and click on another.  All items in between will highlight.  To highlight multiple discrete items, hold down ctrl/cmd and click.  Each individual item you click on will highlight.  When you select edit metadata in bulk, the metadata you edit will apply to all highlighted items.

Calibre will separate out what they call the Basic metadata from the Custom metadata, allowing you to tweak your own separately to the default fields.  This should be fairly straightforward, but if you have issues, let me know.

Getting Fic to your Mobile Device

This used to be a lot easier, and I don’t know what happened.  It’s been years since I’ve been able to plug my iPad into my computer and just sync it that way.  Now, they want you to use this Server thing, and I cannot make it work for anything. I’m kind of convinced it doesn’t work at all.

Directly via Dropbox

As long as you have set up Calibre and FanFicFare to put your fic in an accessible place, and download fic in a format that is readable on your device, you are pretty much there.  You may not even need to go any further.  If you’ve used Dropbox, and your mobile device is capable of using a Dropbox app, you’re done!  You can go to your Calibre folder, and start navigating through all of the folders in there and start reading.  It’s a bit unwieldy though, and that might not be for you.

Another option is to not use Dropbox as your main library, in which case you can click that Calibre icon up top, and move your library elsewhere, and then copy books individually to Dropbox as you need them.  I really dislike the way Calibre structures its library, but I get it.  It’s structured like a library, by author.

For Kobo users, here’s some information and troubleshooting on how to get Dropbox working.

Dropbox via Companion App

On iOS and Android, there are a variety of companion apps, some paid, some free.  On iOS, I use Calibre Sync, which was about $4 or so.  It talks to Calibre and sees all of my custom metadata, and then pulls the ePubs from my dropbox and puts them into my Books app.  I see it as a $4 convenience fee to keep me from having to dig through hundreds of folders.  If you’re on iOS or Android, look around and find the app that works best for you.

Drag and Drop

You don’t need Dropbox for this to work, so if you don’t have it, this is the one for you.  If you’re on iOS, you’ll use iTunes for this.  If you’re on Android or any other device, I can’t offer as much help, because I don’t know how those systems work.  Plug your device into your computer, or use Wifi Sync if your set up supports it.  Go to your Calibre folder where all your books are, and drag the folders into iTunes or wherever they go on your specific device.  Especially if you’re on iOS, you might want a better eBook reader than the one that’s built in, because you won’t be able to see any of your metadata.  In fact, I have just tested and the Books app cannot even see the metadata that AO3 imports, much less the custom metadata.  But if you just want to read, and do all of your organising on Desktop, this works.

I need a different file type than what FanFicFare offers

Not a problem!  The reason most people are likely familiar with Calibre is eBook conversion.  Like the metadata above, you can convert individually or by bulk.  Select the books you want to convert, and click the Convert Books icon in the menu.  You’ll get a big, giant window this time with a whole bunch of options, but the main one you need to focus on is a tiny little button in the top right labelled Output Format.  Click that, and you’ll get a whole list of formats that FanFicFare doesn’t offer.  Select the one you need and click OK.

If know you are going to need to convert your files, I would recommend not changing FanFicFare’s download options.  It will be set to ePub by default, which convert fairly nicely to a wide variety of other formats.

Misc Notes

A few things of import that seem worth pointing out:

  • Your custom columns in the tag browser will all have the same ugly icon unless you change them.  I went to emojipedia and got some cute emoji for mine.  You can use whatever you like.
  • Tags in the book details pane are always listed alphabetically, which is not always helpful if authors use them to build off of one another.  However, if you get confused, like I do, you can always click on the archiveofourown.org link in the IDs field, which will take you to the fic (it looks like it goes to the site’s home page, but it does go to the fic).
  • Anonymously-posted fic will populate as being posted by an Anonymous author unless it is your own and you have given Calibre your own credentials.  Then it will know that you posted the thing.  I got jumpscared by this and I did not like it, because I forgot that you have to sign into the plugin.
  • There is a tiny, almost invisible button in the lower right that says Layout.  Click it!  There are loads of options hidden in there that are actually useful and user friendly!
  • You truly cannot break anything.  If something goes a bit funny, delete what you’ve done, fix the messed up setting, and re-import it.  I have re-imported my entire library about eight times, because I keep fucking around and finding out.  But that’s how you learn.  I’m on the discord and can (hopefully) answer any questions you have.