情况

我有两台服务器，生产和开发。在生产服务器上，有两个应用程序和多个 (6) 个数据库 (MySQL)，我需要将它们分发给开发人员进行测试。所有源代码都存储在GitLab中在开发服务器上，开发人员只能使用此服务器，无法访问生产服务器。当我们发布应用程序时，master 登录到生产环境并从 Git 中 pull 新版本。数据库很大(每个都超过 500M，并且还在增加)，我需要尽可能简单地将它们分发给开发人员进行测试。

可能的解决方案

• 在将数据库转储到单个文件的备份脚本之后，执行将每个数据库推送到其自己的分支的脚本。如果开发人员想要更新他的本地副本，他可以 pull 其中一个分支。

  发现这个无法正常工作。

• 生产服务器上的 Cron 每天保存二进制日志并将它们推送到该数据库的分支中。因此，在分支中，有每日更改的文件，开发人员 pull 他没有的文件。当前的 SQL 转储将以另一种方式发送给开发人员。当存储库的大小变得太大时，我们将向开发人员发送完整转储并刷新存储库中的所有数据并从头开始。

问题

• 解决方案可行吗？
• 如果 git 正在向/从存储库推送/pull ，它是上传/下载整个文件，还是只是更改它们(即添加新行或编辑当前行)？
• Git能管理这么大的文件吗？不能。
• 如何设置存储库中保留的修订数量？与新解决方案无关。
• 有没有更好的解决方案？我不想强制开发人员通过 FTP 或类似的方式下载这么大的文件。

最佳答案

2017 年更新:

Microsoft 正在为 Microsoft/GVFS 做出贡献: 一个允许 Git 处理“the largest repo on the planet”的 Git 虚拟文件系统
(即:Windows 代码库，大约有 350 万个文件，当 checkin Git 存储库时，会产生大约 300GB 的存储库，除了数千个 pull 请求外，还会在 440 个分支中每天生成 1,760 个“实验室构建”验证构建)

GVFS virtualizes the file system beneath your git repo so that git and all tools see what appears to be a normal repo, but GVFS only downloads objects as they are needed.

GVFS 的某些部分可能会贡献到上游(Git 本身)。
但与此同时，all new Windows development is now (August 2017) on Git .

2015 年 4 月更新:GitHub 建议:Announcing Git Large File Storage (LFS)

使用 git-lfs (参见 git-lfs.github.com )和支持它的服务器:lfs-test-server ，您只能将元数据存储在 git 存储库中，而将大文件存储在其他地方。 每次提交最多 2 Gb。

参见 git-lfs/wiki/Tutorial :

git lfs track '*.bin'
git add .gitattributes "*.bin"
git commit -m "Track .bin files"

原答案:

关于大文件的git限制是什么，你可以考虑 bup (在 GitMinutes #24 中有详细介绍)

design of bup 强调限制 git 存储库的三个问题:

• 大文件(xdelta for packfile 仅在内存中，不适合大文件)
• 大量文件，这意味着每个 blob 一个文件，而且 git gc 一次生成一个 packfile 很慢。
• 巨大的包文件，包文件索引无法从(巨大的)包文件中检索数据。

处理大文件和xdelta

The primary reason git can't handle huge files is that it runs them through xdelta, which generally means it tries to load the entire contents of a file into memory at once.
If it didn't do this, it would have to store the entire contents of every single revision of every single file, even if you only changed a few bytes of that file.
That would be a terribly inefficient use of disk space, and git is well known for its amazingly efficient repository format.

Unfortunately, xdelta works great for small files and gets amazingly slow and memory-hungry for large files.
For git's main purpose, ie. managing your source code, this isn't a problem.

What bup does instead of xdelta is what we call "hashsplitting."
We wanted a general-purpose way to efficiently back up any large file that might change in small ways, without storing the entire file every time. We read through the file one byte at a time, calculating a rolling checksum of the last 128 bytes.

rollsum seems to do pretty well at its job. You can find it in bupsplit.c.
Basically, it converts the last 128 bytes read into a 32-bit integer. What we then do is take the lowest 13 bits of the rollsum, and if they're all 1's, we consider that to be the end of a chunk.
This happens on average once every 2^13 = 8192 bytes, so the average chunk size is 8192 bytes.
We're dividing up those files into chunks based on the rolling checksum.
Then we store each chunk separately (indexed by its sha1sum) as a git blob.

With hashsplitting, no matter how much data you add, modify, or remove in the middle of the file, all the chunks before and after the affected chunk are absolutely the same.
All that matters to the hashsplitting algorithm is the 32-byte "separator" sequence, and a single change can only affect, at most, one separator sequence or the bytes between two separator sequences.
Like magic, the hashsplit chunking algorithm will chunk your file the same way every time, even without knowing how it had chunked it previously.

The next problem is less obvious: after you store your series of chunks as git blobs, how do you store their sequence? Each blob has a 20-byte sha1 identifier, which means the simple list of blobs is going to be 20/8192 = 0.25% of the file length.
For a 200GB file, that's 488 megs of just sequence data.

We extend the hashsplit algorithm a little further using what we call "fanout." Instead of checking just the last 13 bits of the checksum, we use additional checksum bits to produce additional splits.
What you end up with is an actual tree of blobs - which git 'tree' objects are ideal to represent.

处理大量文件和git gc

git is designed for handling reasonably-sized repositories that change relatively infrequently. You might think you change your source code "frequently" and that git handles much more frequent changes than, say, svn can handle.
But that's not the same kind of "frequently" we're talking about.

The #1 killer is the way it adds new objects to the repository: it creates one file per blob. Then you later run 'git gc' and combine those files into a single file (using highly efficient xdelta compression, and ignoring any files that are no longer relevant).

'git gc' is slow, but for source code repositories, the resulting super-efficient storage (and associated really fast access to the stored files) is worth it.

bup doesn't do that. It just writes packfiles directly.
Luckily, these packfiles are still git-formatted, so git can happily access them once they're written.

处理巨大的存储库(意味着大量巨大的包文件)

Git isn't actually designed to handle super-huge repositories.
Most git repositories are small enough that it's reasonable to merge them all into a single packfile, which 'git gc' usually does eventually.

The problematic part of large packfiles isn't the packfiles themselves - git is designed to expect the total size of all packs to be larger than available memory, and once it can handle that, it can handle virtually any amount of data about equally efficiently.
The problem is the packfile indexes (.idx) files.

each packfile (*.pack) in git has an associated idx (*.idx) that's a sorted list of git object hashes and file offsets.
If you're looking for a particular object based on its sha1, you open the idx, binary search it to find the right hash, then take the associated file offset, seek to that offset in the packfile, and read the object contents.

The performance of the binary search is about O(log n) with the number of hashes in the pack, with an optimized first step (you can read about it elsewhere) that somewhat improves it to O(log(n)-7).
Unfortunately, this breaks down a bit when you have lots of packs.

To improve performance of this sort of operation, bup introduces midx (pronounced "midix" and short for "multi-idx") files.
As the name implies, they index multiple packs at a time.

关于带有大文件的 Git，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/17888604/