Insync is a commercial option for Google Drive that apparently alliviates many of the problems including Linux, symlink support, and ignore lists.No Linux client (cannot use on CoCalc).Reasonably fast syncing (slower than Dropbox).This may be a more attractive solution once G-Suite for Education is setup at WSU. Some of these may be resolved using Insync (commercial one-time-fee). git directories (these should be typically excluded).īecause it offers more space, we use this for syncing data with experimentalists, but it also has a range of issues. Excluding a directory from syncing deletes it locally so you have to go through a convoluted procedure of copying it (you can't move it or you wont be able to exclude the non-existent directory), excluding it, and then restoring the contents.(Allows you to have your actual directory elsewhere on your computer rather than have to keep it in the drive.) Available Linux client (works on CoCalc).Automatic syncing with conflict resolution.The main issue is lack of space relative to Google Drive and One Drive, as well as issues with selective sync: Dropbox ¶Ĭurrently our preferred tool for sharing. We run simulations on our local machines, office desktops, a local HPC cluster Kamiak and online using CoCalc. WSU provides 1TB of storage through Microsoft One Drive for all students and faculty, so this would be a natural storage too, but few use it yet.

Some of the collaborators are used to using Dropbox or Google Drive, which work for syncing, but have issues (listed below). We need to share source code, experimental data, papers, plots, and simulation data on a regular basis. I run a research group at WSU with ~10 collaborators. To specify this folder, I would use the remote_id option. This be in something like Team Drives/Project 1/git-annex/ hosted at WSU. google-team-wsu: Typically I share data through a google Team Drive.I would use a prefix such as mmfbb.gpe_data to organize the data within a folder that points to where I host the master repo. google-wsu: My generic git annex repo in My Drive/git-annex/ at WSU.For example, the default documentation uses names such as 'google' which are fine for testing, but should be replaced by something more specific later. You must make sure that your remotes have useful names so that you and your collaborators can find them. which do not run git.Įach remote has a name that is stored in the git repository. These are managed within your repository but can point to many different places, such as folders, Google Drive, etc. The utility of git-annex comes from storing data on remotes. This explains the difference between sync (which syncs meta-data but not the file contents) and sync -content. The former includes autostart information about which repositories should be synced. Configuration information is stored in ~/.config/git-annex and.git folder for that annex) appear in the Dashboard. You can create other repositories from the GUI, but they will not appear in the Dashboard - only the repositories managed by the current 'annex' (i.e. This seems to open the annex in the current folder or the last used annex. For a GUI you can run git-annex-webapp.All of the annex information (log files, repository information, etc.) is stored in the. You can use the annex as both a git-annex and a regular git repo if you choose. Thus, if you do not initialize the git repository, git status will return fatal: This operation must be run in a work tree. Note: the annex information is separate from regular git information. Each 'annex' is a git repository where the annex information is stored under the.

Here is my current understanding of git-annex and the git-annex assistant.

Utilize cloud storage options including Google Cloud, Dropbox, Microsoft One Drive both as redundant backups, but also as a mechanism for sharing data with others who need to be able to use only once of these services.

laptops) if memory on that device is limited.

Allow only a subset of data to be stored on any particular device (esp.

Data shared across many platforms: HPC clusters, laptops, desktops, Mac, Windows, Linux, CoCalc, etc.

Here we explore git-annex as a mechanism for replacing and interacting with Dropbox, Google Drive, One Drive etc. The annex keeps track of where the file actually resides (which may be in a different repository, or on another compute) and allows you to control the file (renaming, moving, etc.) without having to have the actual file present. The idea is to store the information about the file in a git repository that can be synchronized, but to store the actual data separately. Git-annex is a tool for managing large data files with git.

Git annex dropbox