Mailspring-Sync

This repository contains the source code for Mailspring’s sync engine, a native C++11 codebase that targets Mac, Windows, and Linux. It leverages the MailCore2 IMAP/SMTP framework and uses sqlite3 to store mail data in a JSON format.

This codebase is GPLv3 licensed. You are welcome to copy, modify, and distribute this code for private or public use, but all derivative works must be open-sourced with a GPLv3 license. If you make improvements to mailsync that would benefit the upstream project, pull requests are greatly appreciated. If you have questions about licensing Mailsync or whether your use is acceptable, please email me at ben@foundry376.com.

The names of the Mailspring or Mailspring Mailsync project nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

Contributing

Mailspring-Sync is entirely open-source. High-quality pull requests and contributions are welcome! When you’re getting started, you may want to join our Discourse so you can ask questions and learn from other people doing development.

Core Concepts

Each instance of Mailsync runs email, contact, and calendar sync for a single account. Mailspring runs one mailsync process for each connected email account, and automatically starts/stops/restarts them as necessary. This design means that auth failures, connection errors, etc. can simply terminate the process.

Mailsync can be run on a standard command line, which makes it easy to test separate from Mailspring. It requires an account and Mailspring ID, and these can be provided either as arguments or as newline-separated data on stdin. If you’re developing in Xcode or in Visual Studio, you can configure the debugger to pass these arguments automatically so testing is easy.

When Mailsync modifies mail data, it emits all modified objects to stdout as newline-separated JSON objects. This restricts the kinds of queries that can be used to modify the database but is critical for providing data to the Mailspring UI, which is built on reactive data patterns and uses observable queries. These object changes are broadcast within the Electron app and cause observable queries backing various views to update and display new data.

Mailsync accepts “tasks” via stdin as newline-separated JSON objects. Tasks are added to a queue table in sqlite (so task completions appear as object modifications on stdout), and are divided into two parts: an immediate “local” operation and a “remote” operation that requires network access and may be retried. This design is necessary for Mailspring, because it’s reactive design means no UI changes occur until the task executes “local” changes and modifies the database.

Sync Approach

Mailspring uses a fairly basic syncing algorithm, which runs on two threads with two open connections to the mail server. Within each thread, work is performed synchronously.

• Background Worker: Periodically iterates over folders and (depending on the supported IMAP features) uses CONDSTORE / XYZRESYNC to check for mail or performs either a “local” or “deep” sync of part of the folder’s UID space.

• Foreground Worker: Idles on the primary folder and wakes to syncs changes. Also wakes to perform other tasks, like fetching message bodies the user clicks.

Sync Design Considerations

• Table Design: Mailspring’s approach to SQLite is similar to CoreData’s: A data column stores a full JSON representation of the model, and additional columns contain copies of the individual fields so the table can be queried. This “fat” approach means fields can be added easily (to the JSON) and migrations are only necessary when adding new queryable columns. The only known downside is that the increased row size caused by the duplication of data into columns and the non-fixed row size makes SCAN queries slower. In retrospect we may have chosen a different approach.

• Message Contents: Mailspring only fetches the contents of the last three months worth of email (configurable in the SyncWorker). For older emails, it fetches only the headers necessary to display subject lines and build stable IDs.

• Stable IDs: Mailspring hashes message headers to create a stable ID for each message. In rare cases, two messages in an account can have the same ID and Mailspring will only show one.

• Metadata: Mailspring allows you to attach arbitrary metadata to threads and messages, and syncs this data to id.getmailspring.com so it can be shared between computers and modified server-side (for read receipts, etc). Metadata objects are stored directly on the corresponding models and modifications to metadata are broadcast to the Mailspring application as modifications to their parent objects. A separate thread in mailsync uses libcurl to listen to a streaming endpoint for metadata events.

• Gmail: Gmail’s IMAP interface makes extensive use of “Virtual Folders” for labels. Mailspring ignores all of these, and only syncs Spam, All Mail and Trash. It then uses the X-GM-LABELS extension to add labels.

File truncated at 100 lines see the full file

Contributing to Mailspring-Sync

We’re always looking for help improving the stability, performance, and functionality of Mailspring-Sync. There are a few ways to get involved:

Contributing Code

One of the best ways is to grab a bug report, sync issue, or feature suggestion that has been marked accepted and dig in. We mark anything specific to Mailspring-Sync with mailsync. You can especially look for issues marked as help-wanted or low-hanging-fruit.

Please be wary of working on issues not marked as accepted. Just because someone has created an issue doesn’t mean we’ll accept a pull request for it.

Pull requests

We welcome PRs that fix bugs, improve functionality, or clean up the code, but not all at once. It’s easier to merge PRs that focus on one thing. It’s a lot easier to feel safe merging a big PR that moves things around if we know the author was ONLY moving things around, and not changing the business logic at the same time.

The first time you submit a pull request, a bot will ask you to sign a standard, bare-bones Contributor License Agreement. The CLA states that you waive any patent or copyright claims you might have to the code you’re contributing. (For example, you can’t submit a PR and then sue Mailspring for using your code.)

Triaging Issues

Another way to help is to answer questions, or to triage bug reports and sync issues. If you’re good at testing and addressing issues, we’d love your help!

Filing Issues

If you have a bug to report or a feature to suggest, you may do so on our Discourse. Please carefully follow all instructions in the template for the given type of issue you’re reporting.

We also ask that you search for existing issues, including closed ones! We’re in the process of migrating. If someone has already reported a bug or requested the feature you have in mind, vote for it at the top. Higher voted issues are more likely to be addressed.

For bugs, please verify that you’re running the latest version of Mailspring (and thus, Mailsync). If you file an issue without providing detail, we may close it without comment.

Under no circumstances should you report an issue via GitHub. The GitHub Issues feature is exclusively for Mailspring contributors to track tasks which have been diagnosed, accepted, and scheduled on the roadmap.

Build and Run From Source

(We’ll be adding this section soon! Please stay tuned.)

Code of Conduct

In order to keep the conversation clear and transparent, please limit discussion to English and keep things on topic with the issue. Be considerate to others and try to be courteous and professional at all times.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.