Archive for the ‘Uncategorized’ Category

Arq 4 SFTP critical issue

Tuesday, April 8th, 2014

Please read this if you’ve been backing up to an SFTP server with Arq.

Potential Incomplete SFTP Backup Data

If you used Arq versions older than 4.1.1 to back up to an SFTP server, you may be affected by an issue that was fixed in Arq 4.1.1.

Arq 4.0, 4.0.2, 4.0.3, and 4.1 could potentially have written incomplete data files to your SFTP server. This issue would not show up until you tried to restore your files, and that is obviously not the time you want to find out that your backup data aren’t complete!

What Is Affected

Only backups to SFTP created by the above versions of Arq are affected. Backups to S3, Glacier, Greenqloud, DreamObjects, or other S3-compatible servers are not affected.

Steps to Resolve the Issue

If you used Arq versions 4.0, 4.0.2, 4.0.3 or 4.1 to back up to an SFTP server, please do the following:

  1. Open Arq and pick “Preferences” from Arq’s menu to open the preferences window.
  2. Select the “Targets” tab in the preferences window.
  3. Select your SFTP target and click the “-“ button to remove it.
  4. Click the “+” button to add your SFTP target.
  5. In the main window, add your folders to the SFTP target as you did originally.
  6. Click the triangle next to “Other Computers”; select your computer in the list, and click the button to delete the now-unused SFTP data.

Arq will back up your files again from scratch. I sincerely apologize for the inconvenience, which I realize is potentially a massive one if you’ve already uploaded a large amount of data.

The details of the issue and how it’s been fixed are explained below.

- Stefan


Details of the Issue and the Fix

The issue was that Arq would write a file to the SFTP server, and if the connection was lost and never recovered, or Arq was restarted, an incomplete file would be left on the SFTP server. Arq assumes that objects on the SFTP server are complete, just as it does for objects written to S3 or Glacier. The difference with S3 and Glacier is that those APIs are “atomic”; either the file is completely stored, or the transaction fails. With SFTP there’s more than one transaction — Arq opens the file, writes many chunks of data to the file, and then closes the file.

The issue never cropped up during a lot of internal testing and several weeks of heavy beta testing. I think this is because SFTP (SSH) re-establishes the network connection whenever it’s lost, and the writes eventually succeed. In some conditions however, such as if the app (or the computer) is restarted, the file could be only partially written.

To resolve the issue, starting with version 4.1.1 Arq writes each file to a temporary directory called “temp” on the SFTP server. The file has a random name (a UUID). When the file is completely written, Arq then moves the file to its final resting place. The move is an atomic operation, so there’s no risk of a partially-written file in the final resting place.

Why I used Java servlets instead of Rails

Monday, November 4th, 2013

Back in 2010 I set up an online store for Arq, and it’s been working well ever since then. It’s based on the excellent Potion Store, which was generously made open source by Andy Kim (thanks Andy!). Potion Store is a Ruby on Rails app. When I first used it, I didn’t have to modify anything about it, which was good because I knew nothing about Rails.

A few years later I wanted to offer discounted multi-license SKUs, so I had to modify the app. I had my Rails book and I walked through the code, or more often the lack of code, since the framework did almost everything by default. In 3 days I learned enough Rails to complete my changes. It worked great, and I wrote almost no code. It was a profound experience — I had never written so little code and gotten so much functionality.

A month ago I needed an online store so people could purchase licenses for Filosync, my new file sync and share product. An obvious option would be to reuse the Potion Store code again. But I decided to write my store from scratch using Java servlets, MyBatis and JSPs, for several reasons.


One reason is speed of change of the framework. Google is nicely tuned for answering programming questions and troubleshooting error messages, but for Rails it was harder for me to get clear answers to my questions. I was never sure which version of Rails was required, or which patch to which module I had to add to my environment. The Java stuff is older and more stable, which makes it easier for me to deal with. Trailing-edge technology FTW! :)


Another reason for choosing Java is obviousness. I was amazed by the Rails approach and how much the framework did for free, without my having to write any code. On the other hand, I felt I had to remember all the things the framework did since there was nothing on the screen showing me what would happen. If I worked with Rails every day I imagine this wouldn’t be a problem, but I don’t work with Rails every day. With the Java approach there’s a lot more typing, and more code means more bugs, but I think it’s easier to follow what’s going on.

I wish there were a Rails IDE that would somehow show me all the things the app is really doing, not just my code — something like the “Show Paragraphs” function in word processors that shows all the hidden characters, except in this case it would show all the hidden functionality.

So, did I do the right thing?I’d like to think that I made the right choice for sound technical reasons, but it could be that Rails was just too far outside my personal comfort zone.

Maybe the problem is just that I prefer libraries to frameworks for this stuff. Java is very simple but it also has everything under the sun if you need something. I just create very simple servlets and then include the libraries I need for database communication, sending mail, and talking to Braintree.

Or maybe the real point of this blog post is it’s hard to take over someone else’s code base and make it your own :)

P.S.  The other day I saw a graph on Twitter showing a decline in Ruby and increase in Java. So I guess I’m on trend?

Please Build a MIME/IMAP-Backed Note-Taking App

Tuesday, October 29th, 2013

There are zillions of note-taking apps available today. Some of them are excellent. I don’t use any of them because my notes/ideas get stuck in them and I can’t get my notes back out. My fear (possibly unfounded) is that I invest months or years storing all thoughts in an app and one day the app stops working, or the app developer stops supporting it, or the app loses my data, or the app morphs into something I don’t want.

I think the apps-contain-data approach used currently by Apple among others is backwards. Data live longer than apps. App should work on data but not contain the data.

I want an open data format for my notes, and an app that helps me manipulate my notes. I think MIME messages and IMAP are perfect for this. I wish someone would build a note-taking app that used MIME and IMAP as its database. Why? Here are some reasons:

  • Every computer and device already has a free IMAP client: the mail app. Lots of free web-based IMAP clients exist as well.
  • IMAP data stores are sync’d across devices.
  • Everyone has already configured their email, so they already have a sync’d data store set up. There’s no need to get your users to sign up for iCloud or Dropbox or anything else.
  • MIME messages are multimedia, and all my devices already have an app that can create, read, edit and delete multimedia MIME messages — the mail app.
  • Everyone already manages the lifecycle of their emails. They protect them, they migrate them to new providers, etc.

I wish someone would create an awesome note-taking app that uses MIME and IMAP. It could store the notes in its own IMAP folder. It could store the metadata about each note within the MIME message, and cache all the notes’ metadata locally so that lookups are faster. Whenever I’m on a system that doesn’t have the app installed I can still read my notes because my mail reader can show them to me, and I can create MIME messages and save them in the note-taking app’s folder. The note-taking app could later add whatever metadata it wanted to those items.

My notes are going to outlive any note-taking app. If my notes are in MIME format in my IMAP account, then they’re not trapped in any particular app.

I would build the app myself if I had time, but I’m working on both Arq and Filosync. So I’m throwing the idea out there hoping someone else builds it.

Maybe one of the excellent existing apps could be modified to use my IMAP account as its data store?

5 lessons learned from starting and running an indie software business

Tuesday, September 17th, 2013

It’s been a few years since I’ve had a “real job”, and almost 4 years since I started selling my first product, Arq. I’m amazed at how much I’ve learned through this process. I see now why the conventional wisdom is to “just do it.” Nobody’s going to map out the plan for you. It’s all up to you.

In preparation for an interview I made up a list of lessons learned. Without further ado:

1. People don’t want to change the front end they’re using

Don’t try to replace it. I learned this by spending a year (in 2008) on an application for multiple people (specifically me and my wife) to work on a shared photo collection. I built it because I was tired of plugging the camera into both computers, worrying about duplicates, etc. It had most of the features of iPhoto, plus it automatically synchronized albums of photos among our computers and a web site. No more waiting around for uploads! At the end of 2009 I said to my wife, “Great news! The app is finally ready for us to start using it.” Her reply (she was a little stressed at the moment): “Does this mean I can’t use iPhoto anymore?”

Hmm. If I can’t get my own wife interested in it, that’s not a good sign!

Eventually I realized I should just build something that copied new photos around to both our iPhoto libraries, and that would get us most of the way there. That product is called SyncPhotos. It does 1 thing, it does it well, and it’s been pretty successful so far.

2. Bug-free products make support almost a non-issue

I used to worry that once I got a number of customers, supporting them would take up too much time for me to move the ball forward on new releases and new products. The only strategy I had to counteract this was to ship products that are as simple as possible and as bug-free as possible. It seems to have worked so far. As a side benefit, I found out that people much prefer super-simple products that don’t have bugs :)

3. You don’t need an enhancement-request database

I heard someone from 37 signals once say that they don’t bother tracking enhancement requests because if it’s important enough, people will continually ask for it. It sounded arrogant to me at the time, but he was right. Version 2 of Arq was easy to plan, as was version 3. I just added the items that I most frequently got emails about. If no one is emailing asking for more, that might mean no one cares, which to me would mean moving on to a new project.

4. Super-fast email response adds tremendous brand value

I seem to have developed a great reputation simply by being very responsive to customers. People have tremendous patience and understanding if you just take the time to communicate with them. I try to answer emails immediately, though lately I’ve been trying to do it only in the early morning and late afternoon, so that I can get more programming done.

I don’t understand why any company would outsource customer support. Talking to customers is the only way you’re going to find out what customers want (to buy)!

5. You’ve got to enjoy helping people

If your passion is putting great software out into the world that makes people’s lives better, people can tell. They can feel the love you’ve put into the product, and they’ll come to love it too. They’ll tell their friends. 

But if your passion is just making money, people can sense that as well. They won’t love your products as much, and they won’t tell their friends as much.

There’s so much bad software in the world — you can make a difference by adding some great software. You’re the boss, so there’s nothing stopping you. Just do it!

Arq, SyncPhotos and Duplifinder are Mountain Lion compatible

Sunday, July 29th, 2012

OS X Mountain Lion is out, and all Haystack Software apps are compatible:

Arq 2.8.2: online backup to Amazon S3

SyncPhotos 2.2.2: sync your iPhoto Libraries across computers

Duplifinder 2.1.1: eliminate duplicate photos

If you do happen to run into an issue with any of the above, please email and I’ll address the issue very promptly!

Gatekeeper and Leopard/Snow Leopard Compatibility

Wednesday, July 11th, 2012

Both Arq (our backup app)and SyncPhotos (our photo sync app) were bitten this week by the code-signing gotcha that Daniel Jalkut, Ben Artin, and Chris Suter blogged about.

I had switched to using Apple’s new Developer ID scheme for code signing so that when Mountain Lion comes out people will still be able to use Arq and SyncPhotos. I even watched the WWDC video to make sure I was doing it correctly. But apparently I didn’t watch closely enough. As the 3 blog posts linked above explain, the code signatures produced with the new scheme can only be validated by Macintosh computers that have the necessary information to validate them, namely computers running OS X 10.7 or later.

Both Arq and SyncPhotos launched OK on 10.6, 10.7 and 10.8. But on 10.6 they asked for keychain read permission over and over again. This was because OS X couldn’t validate the apps’ signatures and therefore didn’t recognize them.

If you’re working on a Macintosh app and you want to support 10.6, you’ll need to do some extra steps when building it. If you’re not, then the rest of this blog post will be really boring!

For a full explanation of the issue involved, please read the linked blog posts above. They were a great start for me, but didn’t give precise enough step-by-step instructions. In the hope of saving someone else from the hours of trial and error I wet through, I’ve listed the steps I followed below. It worked for me every time. The one crucial difference from the referenced blog posts I think is in step 2, booting into 10.6 before exporting the designated requirement.

Leopard-Compatible Code Signing Using Developer ID

  1. Build your app using Xcode 4.3. This may be painful if you’ve been building in Xcode 3, but fix up the errors as best you can to get a successful build.
  2. Boot OS X 10.6 (I have it on a second hard drive in my MacBook Pro, which is handy), get the app that you built in step 1, and run the following on it:
        codesign -d -r- <path to app>
  3. Find the line in the codesign output that starts with “designated” and store that line in a text file named designated_requirement.txt in the root folder of your project.
  4. Boot your normal development environment (for me this is 10.7).
  5. Discard all those changes you made to your project in step 1 by doing whatever command your revision control system has for that purpose. 
  6. Open your app’s project in Xcode 3. Don’t set code signing as an Xcode build setting. Instead, add a custom build step at the end that looks something like this:
        codesign -f -s 'Developer ID Application: (your company)' \
        -r $SOURCE_ROOT/designated_requirement.txt \
  7. Build it in Xcode 3.
  8. Verify the app’s signature using the codesign utility:
        codesign --verify -vvv <path to app>
  9. Boot OS X 10.6 again, get the app you built in step 7, and verify it again:
        codesign --verify -vvv <path to app>

    In both cases you should see output like this: valid on disk satisfies its Designated Requirement

Arq 2.6.8 is out

Tuesday, April 3rd, 2012

Arq 2.6.8 is available. To update, pick “Check for Updates” from Arq’s menu.

Please note: Arq 2.6.8 may upload much more than you expect or want it to the first time it backs up. This is due to a bug that I unfortunately created in version 2.6.3. I apologize for this. Please let it finish backing up.

There is no charge from Amazon for data transfer to Amazon, but I realize that re-uploading a lot of data is a long, frustrating experience.

The full story of what happened starting with Arq 2.6.3

There’s a problem with some of S3′s servers, particularly in the us-west-1 region, but also in other regions. Connections just get “stuck”. Customers were complaining heavily about Arq getting “stuck”. I was initially unable to reproduce the problem, but felt I had to do something to fix it. The Apple networking API that Arq was using had no facility for timing out; plus it was very hard to throttle properly. So I changed Arq to use Apple’s older CFHTTP API. 

In my haste to fix this customer problem, I didn’t do enough testing on 2.6.3-4. I screwed up. There was a bug in those releases where some objects were marked as “compressed” but they weren’t actually compressed. So Arq couldn’t restore properly.

What really made things bad was that at some point in my rush to get the issue fixed I accidentally (because of muscle-memory habit) ran the command to publish 2.6.3 in the “update stream” as an official release of Arq, which meant that everyone who happened to check for updates would get this buggy version. I didn’t realize this for several hours. At that point I could only move forward and fix things as quickly as possible.

Since there’s no way for me to tell which objects were compressed, Arq 2.6.5 (and later) ignores all backups made after March 7 by an Arq version older than 2.6.5. I added a “version” to the backup data so that if this sort of thing ever happens again, I can fix it with a less drastic solution.

I’ve also added some “gates” to my release process so that I don’t inadvertently do this again.

Arq 2.6.6 through 2.6.8 contain very small changes — checks for various network errors that were causing Arq to fail the backup instead of retry.

Please let Arq finish doing what it’s doing. It will clean up all the “bogus” S3 data after it successfully backs up everything.

Also, please consider using more than one backup system, as I blogged about in 7 facets of a good Mac backup strategy.

Why versioned backup is essential

Wednesday, February 22nd, 2012

When you’re deciding on your computer backup strategy, make sure it includes a form of “versioned backup.”

A Memory For Your Files

Before personal computers, we did our work on paper. The paper approach has a built-in “memory.” If you rewrite a paper document, the old document sticks around until you take action and throw it away. By default you have versioned backup. Even after you throw it away you could still get it back if you’re willing to dumpster-dive and you do it before the garbage truck arrives.

When we switched to personal computers for documents, we lost that “memory.” When you save a new version of your document on your computer, the old version is immediately obliterated — gone forever! It’s unfortunate because having the old versions can be a huge time saver. When you’ve made a mess of things and you want to start over, when you realize you deleted a file that you still need, or when a file becomes unreadable for whatever reason, going “back in time” to get an old version saves the day.

Incomplete Solutions

The “trash can” in Windows and OS X isn’t a complete solution, because it only holds onto files you’ve deleted. If you open a document, make some changes, and then click “Save”, the previous version of the document doesn’t go into the trash can.

Apple has tried to provide a form of versioned backup through OS X Lion’s Auto Save and Versions feature. But it only works in apps that have added support for that feature, so it’s not a complete solution.


So, make sure you include a product in your backup strategy that keeps versions. Time Machine (Apple’s built-in backup app) does it for local backups. Arq does it for online backups. Dropbox does it to some extent (it keeps a few revisions of each file).

Cloning tools like SuperDuper don’t do it (but they have other advantages like quick recovery time).


Arq 2.5 is out!

Sunday, February 12th, 2012

Arq version 2.5 is now available!

This update includes a new feature that posts a Growl notification when a backup completes. It’s useful for folks who hide the menubar icon; now they have a way to know when the backup completes without launching the Arq app. It’s optional — you can turn it off in the preferences.

There are also several bug fixes.

It’s a free update for all Arq users. Pick “Check for Updates” from the Arq menu to get the update.

As always, full release notes for all Arq versions are on the release notes page.

Arq, SyncPhotos, Duplifinder are Lion compatible apps

Wednesday, July 20th, 2011

OS X Lion is out, and all Haystack Software apps are compatible.

Arq 1.7.7: online backup to Amazon S3

SyncPhotos 2.0.7: sync your iPhoto Libraries across computers

Duplifinder 2.0.3: eliminate duplicate photos

If you do happen to run into an issue with any of the above, please email and I’ll address the issue very promptly!