Xedge>> git-internals>> 返回
项目作者： techgaun

项目描述 ：
An overview of git internals
高级语言：
项目地址: git://github.com/techgaun/git-internals.git
创建时间： 2019-06-21T22:14:36Z
项目社区：https://github.com/techgaun/git-internals
开源协议：Apache License 2.0
下载


git-internals

An overview of git internals

This repo consists of the talk given at PayLease’s Show and Tell on 06/21/2019.

Git has a content-addressable filesystem as the layer which acts as KV store in a way.
You give some content to git and git gives you a 40 character sha1 hash. You can then
use the sha1 hash in the future to talk with git about that content.

Slides - Use with mdp slides.md

Walkthrough

Git aliases/Shell aliases/Global gitignore

.git directory

We create a git repository first and look at the initial tree structure of .git.
Git repo is a directory with .git sub-directory with relevant metadata.

  1. $ git init git_demo
  2. Initialized empty Git repository in /tmp/git_demo/.git/
  4. $ cd git_demo
  6. $ tree .git
  7. .git
  8. ├── branches
  9. ├── config
  10. ├── description
  11. ├── HEAD
  12. ├── hooks
  13. │   ├── applypatch-msg.sample
  14. │   ├── commit-msg.sample
  15. │   ├── fsmonitor-watchman.sample
  16. │   ├── post-update.sample
  17. │   ├── pre-applypatch.sample
  18. │   ├── pre-commit.sample
  19. │   ├── prepare-commit-msg.sample
  20. │   ├── pre-push.sample
  21. │   ├── pre-rebase.sample
  22. │   ├── pre-receive.sample
  23. │   └── update.sample
  24. ├── info
  25. │   └── exclude
  26. ├── objects
  27. │   ├── info
  28. │   └── pack
  29. └── refs
  30.     ├── heads
  31.     └── tags
  33. 9 directories, 15 files
  • .git/config holds local git configuration that applies to the repo we are in.
  • .git/description holds description that is shown by gitweb.
  • .git/HEAD holds pointer/reference to what branch/tag/commit id we are at.
  • .git/hooks holds sample hooks initially and you can create your own.
  • .git/info/exclude holds repo level gitignore that doesn’t go in repo’s .gitignore.
  • .git/objects holds all kind of objects git stores.
  • .git/refs holds all kind of references git makes use of (branch/tag/stash, etc.).
  • .git/logs doesn’t exist initially but gets created as you travel through your git repo. It holds all the logs that
    show up on git reflog subcommand.
  • .git/index doesn’t exist initially but holds information about the staging area.

Git hooks

  • scripts that executes before or after certain events such as: commit, push, receive, etc.
  • pre-commit - usage could be something like running lint or unit tests on files changed. Exits without making commit
    if the pre-commit hook returns non-zero exit code.
  • post-receive - usage could be for pushing code to the production.
  • to enable hooks, overwrite or create one of the scripts in .git/hooks and make it executable.

Git plumbing vs porcelain commands

  • Most of the commands we use on our day to day interaction with git are porcelain commands that are much simpler to
    use. Think of them as the frontend for git with simplified interface.
  • There are another sets of commands that are lower level and can be composed together to form the porcelain commands.
    These commands are called plumbing commands.
  • As we explore further, we will look at some of the plumbing commands here in a bit with example.
  • Some of the plumbing commands we will look at are hash-object, update-index, write-tree, commit-tree and
    cat-file.

Git objects

4 types of Objects:

  • blob (binary large object) - the data we want git to store and version
  • tree - pointers to file names, contents & other trees. A git tree object creates hierarchy between files and
    directories in a git repository.
  • commit - tree of changes together with some additional metadata (like author, commit message, committer, etc.). It
    represents snapshot of the state of the repository.
  • tag - For annotated tags which contains hash of tagged object (usually commits are tagged).

Git references

  • names that point to sha1 hashes.
  • stored in directories inside .git/refs.
  • heads contain branch references.
  • tags contain tag references.
  • remotes contain references on remote urls added.

Git packfiles

  • Git stores objects on disk in so called loose object format initially.
  • It would be inefficient if git kept on storing entire content everytime we make change on a file.
  • Git occasionally packs up several of these loose objects into a single binary file called packfile to save space and
    be more efficient. This allows storing versions of objects in the form of deltas.

Git gc/reflog/fsck

gc

  • performs cleanup and optimizes the repository.
  • several housekeepings such as compressing file revisions, removing unreachable objects, packing refs and pruning
    reflogs & stale working trees.
  • As it relates to packfiles, it gathers up loose objects & places them in packfiles. Also, it consolidates packfiles
    into a single large packfile as necessary.
  • Auto gc happens once in a while as git deems necessary for example when you try to push to remote.

reflog

  • git records what repo’s HEAD is everytime it changes which we call reflog.
  • stored in .git/logs directory.
  • can be useful to recover accidentally deleted branches.

fsck

  • Integrity check of your objects in the database.
  • Often gives us the knowledge of dangling objects that can no longer be reached.
  • Could be potentially useful in cases when we don’t have reflogs.

Working Example

We will run series of commands and see how things work under the hood based on the understanding from information above.
We’ve already created git_demo repository earlier while looking at the tree structure of .git directory.

  1. # lets create a simple text file
  2. $ echo "Hello World" > readme.md
  4. # now we can see what would the sha1 hash of readme.md according to git
  5. # the way it works is, a format of data is created as below and then sha1 hash for that is created
  6. # <type_of_object> <size_of_object><nullbyte><content_of_object> | sha1_hash_function
  7. $ git hash-object readme.md
  8. 557db03de997c86a4a028e1ebd3a1ceb225be238
  10. # we can replicate what git did by doing something like below:
  11. # blob is the type of object in this case
  12. # as you can see below, the hash matches, simple ;)
  13. $ echo -e "blob $(wc -m readme.md | cut -d' ' -f1)\000$(cat readme.md)" | sha1sum 
  14. 557db03de997c86a4a028e1ebd3a1ceb225be238  -
  16. # we ran hash-object which is a git plumbing command
  17. # however that doesn't add our content to object database until we instruct git to do so
  18. # next we will do that
  19. # open a new terminal (or tmux split) with the following command
  20. $ watch -n 0.5 tree .git
  22. # now we will hash the object and ask git to store it in git object database as well
  23. # when we do so, git will create new directory .git/objects/55
  24. # and create file with name `7db03de997c86a4a028e1ebd3a1ceb225be238`
  25. # first two characters of sha1 hash form directory and rest the filenames
  26. # thats how git organizes objects in its objects database.
  27. $ git hash-object -w readme.md
  28. 557db03de997c86a4a028e1ebd3a1ceb225be238
  30. # next we will cat the content of file
  31. $ cat .git/objects/55/7db03de997c86a4a028e1ebd3a1ceb225be238 
  32. xK��OR04b�H����/�I�A�I
  34. # above we see some unrecognizable text and thats not what we saved though
  35. # git saves our content with header + nullbyte + content as we saw earlier
  36. # we can use git cat-file plumbing command to look at the object we just created.
  37. # -p means pretty print the content of that object
  38. $ git cat-file -p 557db03de997c86a4a028e1ebd3a1ceb225be238
  39. Hello World
  41. # and now lets look at the type of the file
  42. # -t means print the type of that object
  43. $ git cat-file -t 557db03de997c86a4a028e1ebd3a1ceb225be238
  44. blob
  46. # now the reason the raw content of object is some sort of gibberish
  47. # is because its stored after running zlib compression
  48. # lets see what it has in there
  49. # as we see next, it stores type of object (blob) and content length (12)
  50. # and actual content separated by nullbyte character.
  51. $ cat .git/objects/55/7db03de997c86a4a028e1ebd3a1ceb225be238 | zlib-flate -uncompress
  52. blob 12Hello World
  54. # now lets see an example of creating a tree out of the object we added
  55. # we have added the object to git object database but it has no idea about
  56. # where and how that should exist in our repository
  57. # before we do that, lets look at our git status
  58. $ g status -s
  59. ?? readme.md
  61. # so there's an untracked file which we will add to git's staging area
  62. # we normally do that via git add readme.md for example
  63. # this time, we will use git update-index plumbing command
  64. # which updates .git/index file (the file that holds staging info)
  65. $ git update-index --add readme.md
  67. # if we run git status, that matches with the fact that readme.md is now in staging area
  68. # the ?? from previous status has changed to A now :)
  69. $ git status -s
  70. A  readme.md
  72. # now we can take a look at our staging area more deeply
  73. # 100644 - 100 means blob (040 means tree) and 644 represents permission
  74. # permission in git looks like unix permissions except its much more limited
  75. $ git ls-files --stage
  76. 100644 557db03de997c86a4a028e1ebd3a1ceb225be238 0    readme.md
  78. # now we can create a new tree with the current state of index
  79. # note that current state of index has readme.md file in staging area
  80. # for this, we use git write-tree plumbing command
  81. # and we get a hash back
  82. $ git write-tree
  83. 3a3aff7fa9639da674465c43fac565c1291f952b
  85. # we can use cat-file to look into the content & type of object that hash represents
  86. $ git cat-file -p 3a3aff7fa9639da674465c43fac565c1291f952b
  87. 100644 blob 557db03de997c86a4a028e1ebd3a1ceb225be238    readme.md
  89. $ git cat-file -t 3a3aff7fa9639da674465c43fac565c1291f952b
  90. tree
  92. # now that we have a tree object that holds a blob object we want to be committed
  93. # we can use git commit-tree plumbing command to create new commit object
  94. # with the tree object we just created
  95. $ echo "initial commit" | git commit-tree 3a3aff7fa9639da674465c43fac565c1291f952b
  96. 86aa1cb0eec333b600d5b8c23c9c95d4983d5e6d
  98. # now lets look at the log of current branch
  99. # we should see new commit we just made
  100. # but for some reason, we don't :(
  101. $ git log --oneline
  102. fatal: your current branch 'master' does not have any commits yet
  104. # so where did that commit go then
  105. # if you search for that object in .git/objects, we do see .git/objects/86/aa1cb0eec333b600d5b8c23c9c95d4983d5e6d
  106. # then why didn't it show up on the git log?
  107. # lets see what data we have in that object
  108. $ cat .git/objects/86/aa1cb0eec333b600d5b8c23c9c95d4983d5e6d | zlib-flate -uncompress
  109. commit 181tree 3a3aff7fa9639da674465c43fac565c1291f952b
  110. author techgaun <coolsamar207@gmail.com> 1561256669 -0500
  111. committer techgaun <coolsamar207@gmail.com> 1561256669 -0500
  113. initial commit
  115. # so we have the data such as tree the commit object was created from,
  116. # author, committer and finally commit message
  117. # now we come back to the same question we had
  118. # why did the git log not show that commit?
  119. # the reason is that this commit is not associated to the current branch
  120. # we only created the commit object so far
  121. # now we can do that using git update-ref plumbing command
  122. # which updates .git/refs/heads/master file among other things
  123. # we could have done: echo 86aa1cb0eec333b600d5b8c23c9c95d4983d5e6d > .git/refs/heads/master
  124. # but git does it in a safer way while handling other side effects as necessary
  125. $ git update-ref refs/heads/master 86aa1cb0eec333b600d5b8c23c9c95d4983d5e6d
  127. # now lets see what happens with git log
  128. # as you will see next, our commit is now part of master branch. Voila!
  129. # we just made a commit to git without using normal commands we are used to with
  130. $ git log --oneline
  131. 86aa1cb (HEAD -> master) initial commit
  133. # now lets create another commit with the earlier commit id as the parent
  134. # we repeat same stuff again, this time we create file with much larger content
  135. $ printf 'Hello World.%.0s' {1..1000} > new.md
  137. # lets check the status real quick
  138. $ git status -s
  139. ?? new.md
  141. # now lets add that file to staging area
  142. # note that we will skip hash-object this time
  143. # and the reason why it still works is because
  144. # update-index goes through the process of hashing all the objects
  145. # while adding them to the staging area
  146. $ git update-index --add new.md
  148. # and if we check status, we see its added to the staging area
  149. $ git status -s
  150. A  new.md
  152. $ git write-tree
  153. c4996cfea245445e4bdb0561bf18e29436568e58
  155. # now lets inspect that tree
  156. # we see that this tree contains complete snapshot of what we have in the git repo
  157. $ git cat-file -p c4996cfea245445e4bdb0561bf18e29436568e58
  158. 100644 blob c7fc1d8f722cc984f6c90f4151de8b250eeb6343    new.md
  159. 100644 blob 557db03de997c86a4a028e1ebd3a1ceb225be238    readme.md
  161. # and now lets make commit object with our newly created tree
  162. # as you will see, we pass part of commit hash from first commit we made
  163. # as you can see, we only passed 7 first characters of hash
  164. # as long as git can resolve the part of hash into an object,
  165. # we can use such short partials of sha1 hash
  166. $ echo "Added new file" | git commit-tree c4996cfea245445e4bdb0561bf18e29436568e58 -p 86aa1cb
  167. fed6ba87e445db5175c628cfecbbd0b83526a54a
  169. # we can do cat-file on commit object as well
  170. # note the parent line in this case
  171. $ git cat-file -p fed6ba87e445db5175c628cfecbbd0b83526a54a
  172. tree c4996cfea245445e4bdb0561bf18e29436568e58
  173. parent 86aa1cb0eec333b600d5b8c23c9c95d4983d5e6d
  174. author techgaun <coolsamar207@gmail.com> 1561258195 -0500
  175. committer techgaun <coolsamar207@gmail.com> 1561258195 -0500
  177. Added new file
  179. # also, lets look at the type of commit object with cat-file
  180. $ git cat-file -t fed6ba87e445db5175c628cfecbbd0b83526a54a
  181. commit
  183. # finally, lets update master ref to this commit
  184. $ git update-ref refs/heads/master fed6ba87e445db5175c628cfecbbd0b83526a54a
  186. # and lets check the git log one more time
  187. # and we see things as expected
  188. $ git log --oneline
  189. fed6ba8 (HEAD -> master) Added new file
  190. 86aa1cb initial commit

Other Examples

We will continue to operate on the above repository we created earlier

gc and packfile

  1. # lets look at the size of .git/objects once
  2. # and as per the output below, we are at around 41K with our git object
  4. $ du -b .git/objects/
  5. 4096    .git/objects/pack
  6. 4224    .git/objects/86
  7. 4096    .git/objects/info
  8. 4150    .git/objects/3a
  9. 4177    .git/objects/c4
  10. 4124    .git/objects/55
  11. 4227    .git/objects/0a
  12. 4255    .git/objects/fe
  13. 4207    .git/objects/c7
  14. 41652    .git/objects/
  16. # and now lets look at the tree of .git directory after all the things we did
  17. # hooks directory is not shown here to preserve space
  18. $ tree .git
  19. .git
  20. ├── branches
  21. ├── config
  22. ├── description
  23. ├── HEAD
  24. ├── hooks
  25. ├── index
  26. ├── info
  27. │   └── exclude
  28. ├── logs
  29. │   ├── HEAD
  30. │   └── refs
  31. │       └── heads
  32. │           └── master
  33. ├── objects
  34. │   ├── 0a
  35. │   │   └── 9c3e68d37858d478ad2692e01126e6851d1c93
  36. │   ├── 3a
  37. │   │   └── 3aff7fa9639da674465c43fac565c1291f952b
  38. │   ├── 55
  39. │   │   └── 7db03de997c86a4a028e1ebd3a1ceb225be238
  40. │   ├── 86
  41. │   │   └── aa1cb0eec333b600d5b8c23c9c95d4983d5e6d
  42. │   ├── c4
  43. │   │   └── 996cfea245445e4bdb0561bf18e29436568e58
  44. │   ├── c7
  45. │   │   └── fc1d8f722cc984f6c90f4151de8b250eeb6343
  46. │   ├── fe
  47. │   │   └── d6ba87e445db5175c628cfecbbd0b83526a54a
  48. │   ├── info
  49. │   └── pack
  50. └── refs
  51.     ├── heads
  52.     │   └── master
  53.     └── tags
  55. 19 directories, 26 files
  57. # Now lets see if we can optimize our repo like git promises to by running gc
  58. $ git gc
  60. # And once again, lets see the size of .git/objects
  61. $ du -b .git/objects/
  62. 5855    .git/objects/pack
  63. 4150    .git/objects/info
  64. 4227    .git/objects/0a
  65. 18328    .git/objects/
  67. # many of the objects are gone as we see above
  68. # and our git object database is down to 18K
  69. # if we look at the tree of .git repo, it will be different now
  70. $ tree .git
  71. .git
  72. ├── branches
  73. ├── config
  74. ├── description
  75. ├── HEAD
  76. ├── hooks
  77. ├── index
  78. ├── info
  79. │   ├── exclude
  80. │   └── refs
  81. ├── logs
  82. │   ├── HEAD
  83. │   └── refs
  84. │       └── heads
  85. │           └── master
  86. ├── objects
  87. │   ├── 0a
  88. │   │   └── 9c3e68d37858d478ad2692e01126e6851d1c93
  89. │   ├── info
  90. │   │   └── packs
  91. │   └── pack
  92. │       ├── pack-5dda0074f5c0745e99fad6c6d639ca69f009091e.idx
  93. │       └── pack-5dda0074f5c0745e99fad6c6d639ca69f009091e.pack
  94. ├── packed-refs
  95. └── refs
  96.     ├── heads
  97.     └── tags
  99. 13 directories, 24 files
  101. # As we see above, we have .git/objects/pack with two files .idx and .pack
  102. # git has optimized our repository and created packfile like we said earlier
  103. # there's git show-index command to which you can pipe .idx file
  104. # I leave that as homework for you to look into that and see what you will see in those

reflog and fsck

  1. # lets move master branch to the first commit
  2. $ git reset --hard 86aa1cb
  3. HEAD is now at 86aa1cb initial commit
  5. # now we don't have the top commit because none of the branches reach to that commit
  6. # if you come back at later point in time, you will not remember sha1 hash
  7. # which means we effectively lost that commit
  8. # imagine that it was not intended action, how could we recover?
  9. # given that our master branch had that commit at some point of time,
  10. # reflog records that meaning we can recover that commit
  11. $ git reflog
  12. 86aa1cb (HEAD -> master) HEAD@{0}: reset: moving to 86aa1cb
  13. fed6ba8 HEAD@{1}: reset: moving to HEAD
  14. fed6ba8 HEAD@{2}: 
  15. 86aa1cb (HEAD -> master) HEAD@{3}:
  17. # here we see the sha1 hash of our commit fed6ba8
  18. # and now we can create new branch from that commit effectively saving us from disaster
  19. $ git branch master-recovered fed6ba8
  21. # lets checkout to the newly recovered branch
  22. $ git checkout master-recovered
  24. # now lets look at the log and we see that we have the lost commit back. Voila!
  25. $ git log --oneline
  26. fed6ba8 (HEAD -> master-recovered) Added new file
  27. 86aa1cb (master) initial commit
  29. # now imagine that our reflogs were gone
  30. # is there a possibility of recovery in such case?
  31. # maybe? fsck may help although its not straightforward, esp. in large repos
  32. # to mimic loss of reflog, lets delete .git/logs
  33. $ rm -rf .git/logs
  35. # and if we check our reflog, its empty
  36. $ git reflog
  38. # lets delete master-recovered branch once again
  39. $ git branch -D master-recovered
  40. Deleted branch master-recovered (was fed6ba8).
  42. # now lets run fsck with --full argument
  43. # --full is default in recent git versions which means you don't have to specify it anymore
  44. # this performs a full fledged database verification
  45. $ git fsck --full
  46. Checking object directories: 100% (256/256), done.
  47. Checking objects: 100% (6/6), done.
  48. dangling commit fed6ba87e445db5175c628cfecbbd0b83526a54a
  50. # and if you look above, we see dangling commit
  51. # which is the commit that got lost in oblivion
  52. # now that we know our dangling commit, we can recover that commit just like earlier
  53. $ git branch master-recovered-new fed6ba87e445db5175c628cfecbbd0b83526a54a

Author