writing

Some Python Code Proofed My Book in 5 minutes

/ Logan Sidwell / Leave a comment

I wrote my book word by word, no AI involved. An editor helped me develop the story and a copy-editor made sure the manuscript was clean. I’ve read my book about a dozen times. Then my layout person gave me the final version of the book and I realized I had to read the whole thing again to check for new errors.

First I did it properly. My eyes were basically blind by the end. But I wanted a second sweep. The thing is, any person asked to do the job will make a mistake. They’ll overlook something. They won’t realize one paragraph is copied over twice or accidentally cut a space between two sentences. What I needed was a perfect sweep. A complete comparison between my original manuscript and the final epub document. The kind of sweep that could only be performed by a soulless machine with an inflexible view of correct and incorrect.

When I’m not writing I’m coding, and this kind of repetitive, detail-oriented, clearly defined task is the perfect fit for a machine. In fact, it was such a perfect fit, the whole process only took an hour.

What Did The Machine Do?

First I defined my requirements. This code was written to spot exactly one type of problem, copy-and-paste mistakes performed by the layout person. It’s not going to spot typos, it’s not going to spot grammar issues, and it’s certainly not going to point out plot holes. This machine is very stupid, but it performs its job to the letter.

Manuscript format: DOCX

Final Book Layout format: EPUB

Goal: Review every sentence in the EPUB and DOCX files and identify any sentence missing from one file that is present in the other, this should capture any omissions, insertions, or errors in the final manuscript. Then, identify if any sentences appear in the same manuscript more than once, this should identify any ‘duplicate chapter’ or ‘duplicate paragraph’ problems.

The complete code will be shown at the end in case you want to use it, but first I’ll walk you through the parts.

Step 1: Parse the DOCX Manuscript

import docx

def extract_text_from_docx(docx_path):
    doc = docx.Document(docx_path)
    full_text = []
    for para in doc.paragraphs:
        if para.text.strip():  # skip empty paragraphs
            full_text.append(para.text.strip())
    return '\n'.join(full_text)

This code is pretty straightforward, it parses the .docx file into paragraphs, joins it all together into one big paragraphless blob.

Step 2: Parse the EPUB Book

This code is almost identical to the DOCX, but EPUB has a lot more nuance to its data-types. We have to ensure we only retrieve the actual text items, and parse them out of html into plain-text. Then we join it all together in one big wall of book.

import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup

def extract_text_from_epub(epub_path):
    book = epub.read_epub(epub_path)
    text_content = []

    for item in book.get_items():
        if item.get_type() == ebooklib.ITEM_DOCUMENT:
            soup = BeautifulSoup(item.get_content(), 'html.parser')
            # Remove scripts and styles
            for tag in soup(['script', 'style']):
                tag.decompose()
            text = soup.get_text(separator=' ', strip=True)
            if text:
                text_content.append(text)

    return '\n'.join(text_content)

Step 3: Split the book-blobs into sentences

This part uses a tool called the Natural Language Toolkit (NLTK). Sometimes what NLTK considers a sentence is a little funny, like it’ll join two sets of short quotes together. But we cannot allow perfect to be the enemy of good, so as long as NLTK is responsible for both sentence splitting procedures, the final outputs should be identical.

import nltk
from nltk.tokenize import sent_tokenize
nltk.download('punkt')
nltk.download('punkt_tab')

def split_text_into_sentences(text):
    return sent_tokenize(text)

Step 3: Data cleanup

You may have noticed some really long character replacement stuff. Turns out the docx parser picks up a few too many newlines and the epub parser likes directional quotes, so all of that gets replaced with nice, consistent sentencing.

def docx_scan():
    docx_path = "FILENAME.docx"
    text = extract_text_from_docx(docx_path)
    sentences: List[str] = split_text_into_sentences(text)
    
    for i, val in enumerate(sentences):
        sentences[i] = val.replace('\n', ' ').replace('“', '"').replace('”', '"').replace("‘", "'").replace("’", "'").replace("\'", "'")

    return sentences

def epub_scan():
    epub_path = 'FILENAME.epub'
    text = extract_text_from_epub(epub_path)
    sentences: List[str] = split_text_into_sentences(text)

    for i, val in enumerate(sentences):
        sentences[i] = val.replace('\n', ' ').replace('“', '"').replace('”', '"').replace("‘", "'").replace("’", "'").replace("\'", "'")

    return sentences

Step 4: Crawl through the two books

This is a bit of a doozy, but this function essentially crawls through the final book looking for the next sentence in the manuscript. If it doesn’t find it in 10 sentences, it reports the sentence missing and moves on.
Note: The original draft of this post had a different algorithm that failed to account for sentence order. There’s nothing a programmer does more than tinker with their code, but this function is a big improvement on the original, trust me.

def compare_books(manuscript: List[str], final_book: List[str]):
    # We sweep through final_book searching for sentences from manuscrpt
    book_1_pos: int = 0
    book_2_pos: int = 0
    while book_1_pos < len(manuscript):
        found: bool = False
        target_sentence: str = manuscript[book_1_pos]
        for sweep_position in range(book_2_pos, book_2_pos+10):
            if(sweep_position < len(final_book) and target_sentence == final_book[sweep_position]):
                book_1_pos += 1
                book_2_pos = sweep_position
                found = True
                continue
        
        if not found:
            book_1_pos += 1
            if ' - ' not in target_sentence:
                print(target_sentence)

And because of the way the function is written, we can actually crawl through both books the same way.

    epub_sentences = epub_scan()
    docx_sentences = docx_scan()
    # Check the epub file for errors
    compare_books(docx_sentences, epub_sentences)
    # Check the docx file for errors
    compare_books(epub_sentences, docx_sentences)

There are ~8000 sentences in my book. Since the computer reads both copies twice, it’s only about 32,000 operations. A very cheap, less than one second scan for errors.

All the differences are then written out to a file. There were a bunch of false positives. Of the 54 reported omissions, 4 sentences turned out to contain errors, the rest were quirks of the epub format. But finding real errors means it’s working! And it means my layout person did a fantastic job!

Step 5: Check for duplicates

Finally, we do a quick check in both sentence lists for duplicates. The results here reveal my laziness as an author. It turns out I have ~90 non-unique sentences in my book. Most are ‘He said’, ‘She said’, ‘He nodded’, but the strangest one was “Alpha, Golf, Delta, Charlie.” which is a list of squadrons that are referenced in that exact order on two different occasions.

    non_unique_docx = set([x for x in docx_sentences if docx_sentences.count(x) > 1])

    non_unique_epub = set([x for x in epub_sentences if epub_sentences.count(x) > 1])

    print(f"Docx copies: {len(non_unique_docx)}")
    print(f"Epub copies: {len(non_unique_epub)}")

I verified that the total number of non-unique sentences was identical in the DOCX and EPUB formats and moved on.

Conclusion

I always felt a little uneasy about the final version of my book. Even when I had been through it myself, I couldn’t be sure I hadn’t overlooked a massive error. I still can’t be completely sure, but there’s something really reassuring about having a machine do a run-through. When precision is the aim, somehow the passionless report of a calculator is more comforting than a thumbs-up from a professional.

Complete File:

import docx
import nltk
from nltk.tokenize import sent_tokenize
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup
from typing import List, Set

nltk.download('punkt')
nltk.download('punkt_tab')

def extract_text_from_docx(docx_path):
    doc = docx.Document(docx_path)
    full_text = []
    for para in doc.paragraphs:
        if para.text.strip():  # skip empty paragraphs
            full_text.append(para.text.strip())
    return '\n'.join(full_text)

def split_text_into_sentences(text):
    return sent_tokenize(text)

def docx_scan():
    docx_path = "YOURFILE.docx"
    text = extract_text_from_docx(docx_path)
    sentences: List[str] = split_text_into_sentences(text)
    
    for i, val in enumerate(sentences):
        sentences[i] = val.replace('\n', ' ').replace('“', '"').replace('”', '"').replace("‘", "'").replace("’", "'").replace("\'", "'")

    return sentences


def extract_text_from_epub(epub_path):
    book = epub.read_epub(epub_path)
    text_content = []

    for item in book.get_items():
        if item.get_type() == ebooklib.ITEM_DOCUMENT:
            soup = BeautifulSoup(item.get_content(), 'html.parser')
            # Remove scripts and styles
            for tag in soup(['script', 'style']):
                tag.decompose()
            text = soup.get_text(separator=' ', strip=True)
            if text:
                text_content.append(text)

    return '\n'.join(text_content)

def epub_scan():
    epub_path = 'YOURFILE.epub'
    text = extract_text_from_epub(epub_path)
    sentences: List[str] = split_text_into_sentences(text)

    for i, val in enumerate(sentences):
        sentences[i] = val.replace('\n', ' ').replace('“', '"').replace('”', '"').replace("‘", "'").replace("’", "'").replace("\'", "'")

    return sentences

def compare_books(manuscript: List[str], final_book: List[str]):
    # We sweep through final_book searching for sentences from manuscript
    book_1_pos: int = 0
    book_2_pos: int = 0
    while book_1_pos < len(manuscript):
        found: bool = False
        target_sentence: str = manuscript[book_1_pos]
        for sweep_position in range(book_2_pos, book_2_pos+10):
            if(sweep_position < len(final_book) and target_sentence == final_book[sweep_position]):
                book_1_pos += 1
                book_2_pos = sweep_position
                found = True
                continue
        
        if not found:
            book_1_pos += 1
            if ' - ' not in target_sentence:
                print(target_sentence)

def main():
    epub_sentences = epub_scan()
    docx_sentences = docx_scan()
    compare_books(docx_sentences, epub_sentences)
    compare_books(epub_sentences, docx_sentences)

    non_unique_docx = set([x for x in docx_sentences if docx_sentences.count(x) > 1])
    non_unique_epub = set([x for x in epub_sentences if epub_sentences.count(x) > 1])
    print(f"Docx copies: {len(non_unique_docx)}")
    print(f"Epub copies: {len(non_unique_epub)}")


if __name__ == '__main__':
    main()

Boost Your Writing Productivity with Pen and Paper

/ Logan Sidwell / Leave a comment

I write every day. 

This is a big improvement for me. In the past, I’d try to write everyday, but the words wouldn’t end up on the page. When I thought about writing, it felt big. Like an overdue errand, or kicking off a workout. Even when I was in my chair staring at the computer, my fingers wouldn’t tap on the keys. It was stressful work, and all that stress wasn’t even producing any words.

Three changes made all the difference. The brain didn’t want to write, so I figured out how to change its mind.

Pen and Paper

I used to prewrite on a computer. I’d sit down, let my mind wander, and type out every idea that came to my head. The funny thing was, I’d end up with three pages of pre-written dialog, plot points, and outline. But the moment I started writing for real, it all went out the window. The thinking I had done hadn’t stuck.

There is something fundamental about scrawling words onto a page. Even with my terrible handwriting, I can feel it in my head. You can’t just think of an idea and tip-tap the words with paper. Your brain has to take an idea, turn it into a few words, and convert that into muscle-movements. As the ink goes on the paper, the idea takes on a bigger meaning. Here’s a study confirming it.

When you sit down to write, start with pen and paper.

Nothing real, nothing committal. Just loose ideas, whatever comes to mind, an outline that keeps your brain thinking. My poor pre-writing pages are a junkyard of random notes, doodles, and aggressively circled sentences.

I’ve always found the first real sentence to be the scariest. It was like I had dragged my brain out of a sunny park, into a theater spotlight, and yelled “Perform!” from offstage. But for some reason, ever since I started using a pen and paper, my mind always finds its mark. 

Scheduled Time

My writing used to be inspired. By which I mean I’d wait until motivation struck, then charge into my writer’s room and ride that initial surge of inspiration for all it was worth. Now I’m writing 100,000 word novels and the artist’s inspiration doesn’t visit so often.

I’d always tell myself I was going to start at the bottom of the hour, then it’d be 3:02pm, and I’d tell myself I’d start at 3:30pm. Around 8pm, I’d be exhausted. Worn out by the stress of needing to write, and the reality that the writing wasn’t happening.

Then I put a meeting on my phone. Writing: 12:00pm-2:00pm. 

And I showed up.

My phone buzzes a half hour before. As soon as I see it, I finish lunch, do a last check for texts, plug the device in downstairs, and sit down in my writer’s room. On rough days, I even lock the door. The lock isn’t to keep people out, it’s more of a symbolic commitment to being in the writer’s space for the afternoon.

Writing is an open-ended task. A nebulous thing that half-fills your entire schedule. You can start at any time, even late at night if you feel so inclined. It’s like the cable repairman telling you he’ll be there between 8am and 6pm. Technically the day isn’t full, but you sure can’t do anything else.

Do yourself a favor and turn your writing into a meeting. Here’s a few ground rules: It’s okay to be late to a meeting from time to time. It’s okay to be early. Sometimes you sneak in a snack. But the important thing is that you are present and committed for the majority of the time block. Nobody likes a no-show.

And try not to reschedule your writing day-of. It’s easy to let other tasks push your writing around. To feel like it can happen later in the day, or that it isn’t a priority. Scheduling that time out is a statement to yourself. Your writing matters, and that two hour block belongs to you.

In my experience, turning that nebulous cloud of writing into a 2 hour block will give you more time in the day, not less.

Don’t spend your energy planning to write, spend it writing.

Momentum

Writing is tough. There are so many layers to a good scene, so many details to a story. The unfortunate truth is that it’s too much for one brain to remember for long.

If you stop writing for a week, you forget the upcoming scenes.

Stop for a month, you’ll forget the whole story.

Stop for three months and you’ll forget how to write in the first place.

The hardest day to write is your first day back from a break. Before you can prewrite, you have to remember the story all over again. Worse, you’ll have to remember why you liked it in the first place.

I wish it weren’t the case, but the only way to keep yourself from the pains of ‘returning to writing’ all the time is to never stop. There are two approaches here, daily goals and a writer’s streak.

Daily goals are the recognition that the only way to keep your brain thinking about a story is to make sure that story moves forward every time you write about it. I have a daily goal of 1,200 words. The benefit here is that each time I write, I’m about six pages and a scene further into the story. The downside is that writing 1,200 words can sometimes feel like scaling a mountain. If you can reach your goal everyday, it’s useful. If it’s so large it scares you, you risk exacerbating your existing reluctance.

The Writer’s Streak is a whole lot more forgiving. Each day, you only need a single word for the writer’s streak to continue. In time, writing becomes a habit. You wouldn’t throw away a 150-day writer’s streak just because you’re feeling a little tired, right? It’s a tool to keep you writing on the hard days, and thinking about your story every day.

No matter what method you choose, remember the real goal: keep the story moving, and keep your brain thinking about it.

Writer’s Reluctance

The brain likes to be comfortable, too comfortable. Sometimes I think it hates writing. It sure likes to come up with excuses. Errands to run, friends to see, not enough motivation, fear of imperfection. It’s up to each of us to tame our own brain, to force it to write day after day. But it can be done. We just have to be a little tricky.

My First Book was Terrible

/ Logan Sidwell / Leave a comment

In April of 2020 I wrote my first book. By May, I was done. I was pulling 5,000 word days, scrawling ideas on a whiteboard, I’d think of a plot in the morning and write it in the afternoon. It took 6 weeks. At the time, it was thrilling, I was telling myself I was gonna be a published novelist by my mid-20s. I was already shopping which publisher I wanted to use. For half a year, I had a tab open to the Pegasus publishing open submission page, because this book needed to get out into the world.

Right after it was edited.

My first book was a complicated thing. It tried to walk the thin line of a story about people summoning demons, revenge, power politics, and a bunch of witchcraft. And those were supposed to be the good guys. In retrospect, I think the book has good bones, but I didn’t have the skills to tell it.

The first sign something was wrong was when I started editing. Every chapter needed work, every paragraph had to be rewritten. I was basically rebuilding the book from scratch, but when I looked at the second draft, the quality still wasn’t there.

The nail in the coffin was when I shared it with my Mom. This is a kind lady who always finds the positive in things, and she was eager to read it! In preparation for her notes, I told myself she’s gonna say a lot of nice things, but I’d need to keep an ear open for opportunities to improve.

When she got back to me, she only had one note. “There wasn’t a lot of emotion in the book”.

That may not sound that bad, but notes on creative endeavors are weird. If you get a bunch of small, nit-picky notes on moments and characters, it’s a good thing. It’s a sign your reader followed the story and was invested in it. A note telling you your story has no emotion means your reader had no investment. A death sentence for the work.

In the year since, I’ve spoken to other writers about their experiences. Turns out what I had done was a common occurrence. The first book is terrible. We call it the practice book, and I had written one doozy of a practice book.

Lesson 1: Don’t write in a vacuum

I was focussed on the end-goal. Get a book published, present at a conference, win awards. But it takes years to learn the fundamentals of good storytelling. Churning a book out doesn’t make you a better writer, it highlights what you’re already doing. Strengths and weaknesses. If I had been attending writer’s groups, if I had had an editor, if I had posted samples online, they might have caught my errors before I was finished.

They might have told me my first book shouldn’t have 7 POVs. They might have told me books are meant to live in the minds of the characters, not simply describe the sequence of events. They could have told me foreshadowing isn’t just a writer being clever, it’s essential for helping the reading process the events of the book.

Lesson 2: Keep it simple

The plot of my book was compared to Shakespeare’s Hamlet, but not in a good way. Complex, deep characters being handled by a novice with the brush. The only story I had ever written was a hike through a Lovecraftian Jungle, the complexities of Hamlet were a bit more than I could handle. It should have been 100,000 words minimum. I had tried to do it in 52,000. 

It’s advice I’ve heard from film makers, game designers, novelists, and artists. The four-book epic can wait. Start with a story you know you can do well.

Lesson 3: Learn from others

I had a lot of hubris on my first go-around. Ambition is a great thing, but the story I was writing was unlike any other story I had ever read, which meant I was inventing it whole-cloth. If I had searched a little harder, read more deeply, analyzed a few more stories, I might have found a framework from which I could build out my story.

When I pitched my story to an editor, they said, oh it’s kind of like a Jurassic Park for demons. I wish I had heard that feedback before writing the book, because it clarified a lot of what I was trying to do. Some of the story beats in Jurassic Park would fit comfortably into my story and fixed a lot of the awkwardness with the characters. It even clarified what the theme of the story was, capturing demonic beasts was a whole lot like keeping Dinosaurs in a park, something that can only end in disaster, no matter how well-intentioned the characters were.

What I mean to say here is that I could have studied Jurassic Park for good story beats. I could have watched Constantine for a lesson on how to deal with angels and demons. I could have watched The Witch to really elevate the evil aspects. I could have taken notes from the best to better understand my own work.

Conclusion

It’s been five years since I wrote my first book. My second book took a little over a year, and my third book looks like it’ll take a little less than that. I talk to writers as much as I can. I keep my books simple. I read more these days. I put every lesson to practice.

I hope anyone reading this won’t be chased away from writing their first novel. It’s gonna be trouble, but thinking it’s gonna be great is a rite of passage. Write it anyway! Take some big swings! What’s the worst that could happen? For me, I’m just glad that such a painful lesson only took a couple of months to learn.

The fastest way to write your first good book is to write your bad book quickly.

This essay is also available in video form:

My Thirty Seconds in an Action movie

/ Logan Sidwell / Leave a comment

I wrote this up about a week after it happened. April 2023.


Highway at rush hour. Five lanes packed with cars trundling home at sixty miles-per-hour. The vehicle in front of me slowed a little. I was incensed, did this driver not realize they were inconveniencing my day? But their speed dropped and dropped, all the way down to zero. I gripped my steering wheel helplessly, the other lanes speeding past, their wake shaking my little Nisan Versa.

There was movement in the stopped vehicle ahead of me. A man fumbling around in the driver’s seat. I leaned forward and squinted the sun out of my eyes. An emergency? A medical incident? A man finally sick of the same two hour commute every evening? He leaned out of sight. The driver’s side window rattled as if it had been struck. Something was wrong. For half a second, the silhouette of a leg craned back behind the driver’s seat. I wasn’t sure what to make of it. Then the driver’s leg shot forward and pieces of glass sprinkled onto the pavement.

It had all happened in seconds. I didn’t even have a chance to consider moving my vehicle, the other lanes were moving too quick, plus I was enraptured by the action in the car ahead. Through the driver’s side window, a foot receded back into the car, and I was left blind. As I waited, and watched, my ears heard something strange. An oscillating thrum coming from the sky, loud enough to be heard over the traffic, and it was getting louder.

The driver threw himself out the broken window. He was nearly bald, wore a tan windbreaker, and moved fast as lighting. Hardly a second after I saw his feet hit the ground, he turned back to his car and reached for something in the back seat. It was a canvas duffle bag, whose contents will forever remain a mystery. I’d love to tell you that he looked my way, that he considered my car as a viable transport alternative, but the truth is he only shot a brief, furtive look at the sky, at that heavy thrum overhead, then ran for the highway’s concrete wall.

He vanished over the side and I never saw him again.

The whole thing took barely twenty seconds, and I didn’t know what to think. An opening appeared in the next lane, I swerved into it and drove off.

It took about a minute for me to realize I should call 911, long enough for that stranger and his duffle bag to be far in the rear-view mirror. The dispatcher answered flatly, and I blundered my way through an explanation of what happened. I’m pretty sure I repeated myself a few times in the explanation, but dispatcher listened to it all politely. When I finished speaking, she responded, “Yes, well… That’s a very bad man. We’ve remotely disabled his car” She paused, her tone reminded me of a parent trying to teach a child that a stove was hot, “Try to stay away from him. Is that it?” Her nonchalant response left me uncertain. Did she just not care? Was it actually not a big deal? Finally, I spoke.

”That’s all. Thank you” After a moment’s silence, she hung up.

That steady thrum in the sky was a helicopter, I realized that the moment I pulled into the other lane. It was only during the long, quiet drive home that I put together the rest. The fact that there was a helicopter meant the police were well aware of the situation, it also explained the dispatcher’s ambivalence. She had probably heard a hundred calls just like mine. Then there was the broken window. The police must have locked the doors when they disabled the car, and when that stranger realized he was trapped, he was forced to create an escape.

In retrospect, I consider myself lucky. Partially because I wasn’t hurt, but mostly because I had been gifted a front-row seat an authentically cinematic moment in real life. My Thirty seconds in an action movie.