AOMO

AOMO ("Archive Of My Own") is a set of tools designed to streamline the process of downloading works from AO3 ("Archive Of Our Own"), then displaying them in as close as possible to their original formatting on that website.

Why

There are many reasons why you might want to download fanfic from AO3 instead of reading it live off the website. Maybe you expect to be without internet connection for some time and want to continue reading during that time. Maybe you fear that the author of the fic may grow to dislike the fic and delete it, or put it in a hidden collection to make it inaccessible, or that they'll join a cult and replace the text of each chapter with 25k words of raving about how only the true beleivers will be saved.

Regardless of the reason, backing up copies of your favorite fics is usually a good idea. It's lucky, then that AO3 makes it so easy to do - just click the well-labeled button on the top of every AO3 fic page¹, and you'll get this:

Official Download	Original Web Page

As you can see, the official downloads provided by AO3 have very different stylization than the versions on the website. I, personally, find the official downloads rather ugly, but if you disagree then you will have no issue using the official downloads for the vast majority of works on AO3.

Official Download	Original Web Page

Some works, however, make use of an AO3 feature called a 'workskin', which allows authors to write custom CSS Stylesheets to be applied to their work. Many, such as the one shown above, rely very heavily on their workskins, sometimes to the point of being incomprehensible without them. This makes it a fairly big problem that not a single character of a work's workskin is ever preserved in the official download.

All of that is to say: The official downloadable files provided by AO3 will never be sufficient in my eyes, especially not for works meant to use Workskins. AOMO exists to solve that problem.

Screenshot Gallery

Official Download ('Raw')	Original Web Page	AOMO Web Interface

Usage

Every Work on AO3 is assigned a unique ID number which this project calls a WorkID. In each of the example links below, the Work ID is bolded:

https;//archiveofourown.org/works/37130362

https;//archiveofourown.org/works/27665101/chapters/67697743

AOMO includes a script for extracting WorkIDs from AO3 links in bulk, but doing it manually like this is fine for doing a few works at a time, so for now we'll focus on the core parts and come back to the optional scripts later.

Step 1: Download the Raws of your chosen Works

For the purposes of this project, the 'Raw' of a work is the official download of that work, specifically the HTML version. While it is possible to download Raws from AO3 or other sources manually, the preffered way of downloading Raws is to use download.py, as it:

• Automatically saves the files in the location and with the name that later steps expect
• Automatically detects if the Work has a Workskin and saves it as well
• Can easily be used on large batches of works at once
• Automatically adds an invisible HTML comment to the end of the Raw, recording exactly what time the Work was last edited by its author, when the Raw was downloaded, and what version of AOMO it was downloaded with.

To use download.py, feed one or more WorkIDs into its input. To download a single work, you can do echo "<workID>" | ./download.py You can also write multiple WorkIDs into a text file and feed that into the script, either like cat workids.txt | ./download.py or like ./download.py -i workids.txt

Step 2: Extract information from the Raws and store it in the database

While Step 1 and the jobs of several of the optional scripts could reasonably be done manually, Step 2 and 3 can only really be done by the scripts.

Simply feed your chosen WorkID(s) into process.py's input as described above:

echo "<workID>" | ./process.py or cat workids.txt | ./process.py or ./process.py -i workids.txt

For every WorkID given, process.py will find the corresponding Raw (and, when applicable, Workskin), parse it, and load its contents into the database specified by config.ini.

Alternatively, you can use ./process.py --all, which will run process.py on all available Raws, or ./process.py --auto, which will do the same as --all but only for raws that haven't been processed yet.

Step 3: Reconstitute the contents of the database into something readable

Option 1: Web Interface

The "Web Interface" is a set of PHP scripts which work to replicate the AO3 website's interface.

• work.php takes a WorkID and, optionally, a chapter number, and displays it in the same formatting and styling it would've had on AO3.
• listWorks.php lists every Work you've downloaded and processed, including information like the Works' tags and summaries, and provides links to those Works on work.php.
• listSeriesContents.php works like listWorks.php, but with all of the Works in a given Series, instead of every Work in the database.
• listSeries.php lists every Series your downloaded works are part of, and provides links to those Series on listSeriesContents.php The Web Interface also has an optional username & password system, allowing users to track their progress through works across devices, and to use an imperfect recreation of AO3's Site Skin system.

Option 2: export.py

In the future I plan to create an export.py script that will produce .html and .zim files from the database, but for now the only way to view the downloaded works (besides the Raws) is by using the PHP-based web interface.

Optional Scripts

linkparser: Get a list of the WorkIDs of the Works you want to download

Manually extracting the WorkID from the link is fine for just downloading a few works at a time, but gets tedious if you want to do tens or hundreds of works at once. For those and other cases you can also use linkparser.py to convert AO3 links into WorkIDs.

linkparser.py can take not only links to Works, but also links to Series of Works, in which case it'll return the WorkIDs of all the Works in each Series, and links to an AO3 user's profile page, in which case it'll return the WorkIDs of every publicly-available Work that user has authored or co-authored.

To use linkparser.py:

1. Write all of the AO3 links to a file, for example links.txt
2. Feed links.txt into the input of linkparser.py as described above
3. Use the --batch_out (-b) argument to specify what file the WorkIDs should be written to (ie ./linkparser.py --batch-out workids.txt)

All in one command that would be ./linkparser.py --infile links.txt --batch-out workids.txt.

getImages: Download the images linked to in the Works

AO3 is, at its core, a text-only platform - It doesn't host any images or videos uploaded by its users. If you are an author uploading your work to AO3 and you want to include an image, you must first get a link to that image hosted on another website - either by it already having been there or by uploading it to that website yourself - and then put that link to the image in your AO3 Work.

The problem with this is when those images are taken down by the websites they're hosted on and the author either can't or won't re-host the images on another website, in which case the images are simply gone from the most easily accessable version of that Work forever. For some Works this is a minor inconvenience or dissapointment, but some Works rely very heavily on their images and would be made miserable to read or even incomprehensible with the images gone.

To try and fix this problem, AOMO includes getImages.py, which scans through the specified Works to find all links to externally-hosted images, then attempts to download them all to local storage. Then, for all images it was able to successfuly download, it replaces the link to the externally-hosted version of the image with a link to the local version of the image.

To use getImages.py, simply feed the list of chosen WorkIDs into the input as described above (./getImages.py -i workids.txt or cat workids.txt | ./getImages.py), or use the --all argument, which works similarly to process.py's.

mailParser: Extract WorkIDs from AO3 subscription notification emails

mailParser.py can read an "MBox" email storage file like those produced by Mozilla Thunderbird, find every AO3 subscription email (identified by a subject line starting with [AO3] and a sender of do-not-reply@archiveofourown.org), find the WorkID(s) in each email, and write all the found WorkIDs to a file.

Usage examples:

• ./mailParser.py -b workids.txt
• ./mailParser.py -o /dev/null | ./download.py

mailParser.py also has the ability to ignore any emails sent before a certain time. By default, this time will be 24 hours before when the script is ran, but this can be changed with the --since (-s) option.

If --since is provided a string in the format of YYYY/MM/DD-HH:MM:SS, it will use it as an absolute time - so ./mailParser.py -s 2020/01/01-00:00:00 will only accept mail from after midnight on January 1st 2020.

If --since is provided an integer, it will use it as a number of seconds in the past - so ./mailParser.py -s 3600 will only accept mail from the last hour, ./mailParser.py -s 604800 will only accept mail from the last week, and ./mailParser.py -s 31536000 will only accept mail from aproximately the last year.

admin: Assorted administrative tasks

admin.py is a Terminal User Interface app that does assorted tasks too small for deticated scripts but that can't reasonably be done by hand.

As of release version 1.3 it can:

• Add, edit, and delete Users for use with the Web Interface
• Add, edit, and delete Site Skins
• Add and remove WorkIDs from a list of works that will be ignored by download.py
• Remove a given work from the database completely.

More capabilities will be added to admin.py over time as their nessecity becomes clear.

metadata: Fetch the descriptions of Series

The currently poorly named ./metadata.py --all visits the pages for every Series recorded in the database and records various bits of information about them, including any Notes or Descriptions assigned by the authors, the IDs of every Work in the Series, and if the series has been marked as 'complete' or not.

Contact

Source code repository and issue tracker can be found at Codeberg. I, the author, can be contacted by email at cyberpro123@posteo.com, on the fediverse at @Cyberpro123@app.wafrn.net, or on Bluesky at @cyberpro123.at.app.wafrn.net

As of writing this, no person has ever given me reason to beleive they've even looked at the repository for this project, much less been interested in trying to use it. If you do find yourself interested in using AOMO I'd greatly appreciate it if you let me know, especially if you have any questions, complaints, or suggestions about the project.

Compatibility & Requirements

AOMO is currently developed and tested only on Linux. It may be able to work on MacOS or other UNIX-like operating systems, but I have no intention to test this.

This project's development and most of it's testing is done on a machine which runs Debian 13 'Trixie' and uses:

• Python version 3.13.5
• BeautifulSoup4 version 4.13.4
• lxml version 5.4.0
• requests version 2.32.3

admin.py uses the bcrypt module, and was developed with version 4.2.0 The Debian package that provides bcrypt is required by the package that provides requests, so if you're using distro packages there's a good chance you'll have it regardless. Also, as admin.py is the only script to use bcrypt, and none of its functions are strictly nessecary, you can simply choose to not use it.

A basic installation using SQLite for the database requires only the above, but as of version 1.3 AOMO also supports Postgresql and MySQL databases, if you have the following dependancies:

Postgres:

• psycopg (version 3.2.6 used in dev)
• Postgres support in your PHP hosting software (if using WebUI)

MySQL:

• MySQLdb (version 1.4.6 used in dev)
• MySQL support in your PHP hosting software (if using WebUI)

Earlier or later versions of these projects may work, but I haven't done any testing to see what is the earliest version of each that will work.

Setup

1. Download the project, either by downloading a .zip or .tar.gz file from the Releases page, or by running git clone --depth 1 --branch v1.3 https://codeberg.org/Cyberpro123/AOMO.git in your terminal.
2. (Optional) Make a config.ini file to change settings away from the defaults (Change file locations, enable logging in to download AO3-users only works, use non-SQLite databases, etc.). See here for examples.
3. Run install.py to create the directories, initialize the database, and, optionally, install the Web Interface.
   • The contents of the python directory are 'portable', meaning they can go in any directory you like so long as they're all in the same directory.
   • If you're using Postgres or MySQL for your database, and intend to use the databases in the provided docker compose files, it'll give you an error about not being able to connect to the database. Come back and run install.py a second time after Step 4.
     • Yes, run install.py the first time despite the error - if the specified folders don't already exist when you start up the docker containers, docker will create them, but owned by root and unable to be edited by other users.
4. Start a web server with access to the 'webui' directory specified in config.ini (by default ~/Documents/AOMO/Output). I recommend using Docker Compose with one of the provided docker-compose.yamls, but if you already have a preffered web server it should also work as long as it can do PHP.
   • If you edited config.ini before be sure to edit your docker-compose.yaml to match

Restrictions

As part of the image downloading feature, the unicode character U+EA03() is used as a marker for where to insert the URL of the downloaded images. While I can think of no legitimate reason for U+EA03 to appear in a fanfic, if AOMO is used on a Work containing it, it will be replaced with U+EA04.

Performance

See here

Credits

This project used to have ao3_api as a dependancy, but I decided to reimpliment the parts of it that I was using into AOMO directly after I realized its current maintainer seems to have no interest in merging any pull requests to fix its many flaws. Those functions that were used to replace AO3 API took very heavily from its source code, and are marked as such.

To achieve the goal of rendering archived works in as close as possible to their original formatting on the AO3 website, AOMO has long since used copies of the .css files used on said website, by downloading them from the github page for otwarchive, the software AO3 runs on, during the installation process. During the development of release version 1.2, I decided I would need to use a slightly modified version of said css files to impliment a certain feature, and that the most practical way to do so would be to include those files in the AOMO repository. All files in the ao3css directory are verbatim or slightly modified copies of ones from otwarchive, Copyright the Organization for Transformative Works.

As far as I am aware, everything I've done described in the previous two paragraphs is completely legal and compliant with AO3 API's MIT License and otwarchive' GPL v2.0 License. If you have legitimate reason to beleive otherwise, please contact me

The following images are screenshots of the work 'Q. A. B.' by jihnari:

• raw-location.webp
• comparison/qab-tags/web.webp
• comparison/qab-tags/raw.webp
• comparison/qab-tags/aomo.webp
• comparison/qab-tweetybird/web.webp
• comparison/qab-tweetybird/raw.webp
• comparison/qab-tweetybird/aomo.webp

Additionally, the works shown listed in list.webp are the afformentioned 'Q. A. B.' and its sequel 'Q. A. B. (cont.)' by jihnari

Footnotes

This project does not, has never, and will never use the technology that mid-2020s marketing departments are attempting to call "AI".

---

1. See here ↩︎