Part 2: Checkouts, Branching, & Merging¶
Warning
The usage info displayed in the latest
build of the project documentation do not reflect recent changes to the API and internal structure of the project. They should not be relied on at the current moment; they will be updated over the next weeks, and will be in line before the next release.
This section deals with navigating repository history, creating & merging branches, and understanding conflicts.
The Hangar Workflow¶
The hangar workflow is intended to mimic common git
workflows in which small incremental changes are made and committed on dedicated topic
branches. After the topic
has been adequatly set, topic
branch is merged into a separate branch (commonly referred to as master
, though it need not to be the actual branch named "master"
), where well vetted and more permanent changes are kept.
Create Branch -> Checkout Branch -> Make Changes -> Commit
Making the Initial Commit¶
Let’s initialize a new repository and see how branching works in Hangar:
[1]:
from hangar import Repository
import numpy as np
[2]:
repo = Repository(path='/Users/rick/projects/tensorwerk/hangar/dev/mnist/')
[3]:
repo_pth = repo.init(user_name='Test User', user_email='test@foo.com', remove_old=True)
Hangar Repo initialized at: /Users/rick/projects/tensorwerk/hangar/dev/mnist/.hangar
When a repository is first initialized, it has no history, no commits.
[4]:
repo.log() # -> returns None
Though the repository is essentially empty at this point in time, there is one thing which is present: a branch with the name: "master"
.
[5]:
repo.list_branches()
[5]:
['master']
This "master"
is the branch we make our first commit on; until we do, the repository is in a semi-unstable state; with no history or contents, most of the functionality of a repository (to store, retrieve, and work with versions of data across time) just isn’t possible. A significant portion of otherwise standard operations will generally flat out refuse to execute (ie. read-only checkouts, log, push, etc.) until the first commit is made.
One of the only options available at this point is to create a write-enabled checkout on the "master"
branch and to begin to add data so we can make a commit. Let’s do that now:
[6]:
co = repo.checkout(write=True)
As expected, there are no columns nor metadata samples recorded in the checkout.
[7]:
print(f'number of metadata keys: {len(co.metadata)}')
print(f'number of columns: {len(co.columns)}')
number of metadata keys: 0
number of columns: 0
Let’s add a dummy array just to put something in the repository history to commit. We’ll then close the checkout so we can explore some useful tools which depend on having at least one historical record (commit) in the repo.
[8]:
dummy = np.arange(10, dtype=np.uint16)
col = co.add_ndarray_column('dummy_column', prototype=dummy)
col['0'] = dummy
initialCommitHash = co.commit('first commit with a single sample added to a dummy column')
co.close()
If we check the history now, we can see our first commit hash, and that it is labeled with the branch name "master"
[9]:
repo.log()
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e (master) : first commit with a single sample added to a dummy column
So now our repository contains: - A commit: a fully independent description of the entire repository state as it existed at some point in time. A commit is identified by a commit_hash
. - A branch: a label pointing to a particular commit
/ commit_hash
.
Once committed, it is not possible to remove, modify, or otherwise tamper with the contents of a commit in any way. It is a permanent record, which Hangar has no method to change once written to disk.
In addition, as a commit_hash
is not only calculated from the commit
’s contents, but from the commit_hash
of its parents (more on this to follow), knowing a single top-level commit_hash
allows us to verify the integrity of the entire repository history. This fundamental behavior holds even in cases of disk-corruption or malicious use.
Working with Checkouts & Branches¶
As mentioned in the first tutorial, we work with the data in a repository through a checkout. There are two types of checkouts (each of which have different uses and abilities):
`Checking out a branch / commit for reading: <api.rst#read-only-checkout>`__ is the process of retrieving records describing repository state at some point in time, and setting up access to the referenced data.
- Any number of read checkout processes can operate on a repository (on any number of commits) at the same time.
`Checking out a branch for writing: <api.rst#write-enabled-checkout>`__ is the process of setting up a (mutable) staging area
to temporarily gather record references / data before all changes have been made and staging area contents are committed in a new permanent record of history (a commit
).
- Only one write-enabled checkout can ever be operating in a repository at a time.
- When initially creating the checkout, the
staging area
is not actually “empty”. Instead, it has the full contents of the lastcommit
referenced by a branch’sHEAD
. These records can be removed / mutated / added to in any way to form the nextcommit
. The newcommit
retains a permanent reference identifying the previousHEAD
commit
was used as its basestaging area
. - On commit, the branch which was checked out has its
HEAD
pointer value updated to the newcommit
’scommit_hash
. A write-enabled checkout starting from the same branch will now use thatcommit
’s record content as the base for itsstaging area
.
Creating a branch¶
A branch is an individual series of changes / commits which diverge from the main history of the repository at some point in time. All changes made along a branch are completely isolated from those on other branches. After some point in time, changes made in a disparate branches can be unified through an automatic merge
process (described in detail later in this tutorial). In general, the Hangar
branching model is semantically identical to the Git
one; The one exception is that in
Hangar, a branch must always have a name
and a base_commit
. (No “Detached HEAD state” is possible for a write-enabled
checkout). If No base_commit
is specified, the current writer branch HEAD
commit
is used as the base_commit
hash for the branch automatically.
Hangar branches have the same lightweight and performant properties which make working with Git
branches so appealing - they are cheap and easy to use, create, and discard (if necessary).
To create a branch, use the create_branch() method.
[10]:
branch_1 = repo.create_branch(name='testbranch')
[11]:
branch_1
[11]:
BranchHead(name='testbranch', digest='a=eaee002ed9c6e949c3657bd50e3949d6a459d50e')
We use the list_branches() and log() methods to see that a new branch named testbranch
has been created and is indeed pointing to our initial commit.
[12]:
print(f'branch names: {repo.list_branches()} \n')
repo.log()
branch names: ['master', 'testbranch']
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e (master) (testbranch) : first commit with a single sample added to a dummy column
If instead, we actually specify the base commit (with a different branch name) we see we do actually get a third branch. pointing to the same commit as master
and testbranch
[13]:
branch_2 = repo.create_branch(name='new', base_commit=initialCommitHash)
[14]:
branch_2
[14]:
BranchHead(name='new', digest='a=eaee002ed9c6e949c3657bd50e3949d6a459d50e')
[15]:
repo.log()
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e (master) (new) (testbranch) : first commit with a single sample added to a dummy column
Making changes on a branch¶
Let’s make some changes on the new
branch to see how things work.
We can see that the data we added previously is still here (dummy
arrayset containing one sample labeled 0
).
[16]:
co = repo.checkout(write=True, branch='new')
[17]:
co.columns
[17]:
Hangar Columns
Writeable : True
Number of Columns : 1
Column Names / Partial Remote References:
- dummy_column / False
[18]:
co.columns['dummy_column']
[18]:
Hangar FlatSampleWriter
Column Name : dummy_column
Writeable : True
Column Type : ndarray
Column Layout : flat
Schema Type : fixed_shape
DType : uint16
Shape : (10,)
Number of Samples : 1
Partial Remote Data Refs : False
[19]:
co.columns['dummy_column']['0']
[19]:
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint16)
Let’s add another sample to the dummy_arrayset
called 1
[20]:
arr = np.arange(10, dtype=np.uint16)
# let's increment values so that `0` and `1` aren't set to the same thing
arr += 1
co['dummy_column', '1'] = arr
We can see that in this checkout, there are indeed two samples in the dummy_arrayset
:
[21]:
len(co.columns['dummy_column'])
[21]:
2
That’s all, let’s commit this and be done with this branch.
[22]:
co.commit('commit on `new` branch adding a sample to dummy_arrayset')
co.close()
How do changes appear when made on a branch?¶
If we look at the log, we see that the branch we were on (new
) is a commit ahead of master
and testbranch
[23]:
repo.log()
* a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94 (new) : commit on `new` branch adding a sample to dummy_arrayset
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e (master) (testbranch) : first commit with a single sample added to a dummy column
The meaning is exactly what one would intuit. We made some changes, they were reflected on the new
branch, but the master
and testbranch
branches were not impacted at all, nor were any of the commits!
Merging (Part 1) Fast-Forward Merges¶
Say we like the changes we made on the new
branch so much that we want them to be included into our master
branch! How do we make this happen for this scenario??
Well, the history between the HEAD
of the new
and the HEAD
of the master
branch is perfectly linear. In fact, when we began making changes on new
, our staging area was identical to what the master
HEAD
commit references are right now!
If you’ll remember that a branch is just a pointer which assigns some name
to a commit_hash
, it becomes apparent that a merge in this case really doesn’t involve any work at all. With a linear history between master
and new
, any commits
exsting along the path between the HEAD
of new
and master
are the only changes which are introduced, and we can be sure that this is the only view of the data records which can exist!
What this means in practice is that for this type of merge, we can just update the HEAD
of master
to point to the HEAD
of "new"
, and the merge is complete.
This situation is referred to as a Fast Forward (FF) Merge. A FF merge is safe to perform any time a linear history lies between the HEAD
of some topic
and base
branch, regardless of how many commits or changes which were introduced.
For other situations, a more complicated Three Way Merge is required. This merge method will be explained a bit more later in this tutorial.
[24]:
co = repo.checkout(write=True, branch='master')
Performing the Merge¶
In practice, you’ll never need to know the details of the merge theory explained above (or even remember it exists). Hangar automatically figures out which merge algorithms should be used and then performed whatever calculations are needed to compute the results.
As a user, merging in Hangar is a one-liner! just use the merge() method from a write-enabled
checkout (shown below), or the analogous methods method from the Repository Object repo.merge() (if not already working with a write-enabled
checkout object).
[25]:
co.merge(message='message for commit (not used for FF merge)', dev_branch='new')
Selected Fast-Forward Merge Strategy
[25]:
'a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94'
Let’s check the log!
[26]:
repo.log()
* a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94 (master) (new) : commit on `new` branch adding a sample to dummy_arrayset
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e (testbranch) : first commit with a single sample added to a dummy column
[27]:
co.branch_name
[27]:
'master'
[28]:
co.commit_hash
[28]:
'a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94'
[29]:
co.columns['dummy_column']
[29]:
Hangar FlatSampleWriter
Column Name : dummy_column
Writeable : True
Column Type : ndarray
Column Layout : flat
Schema Type : fixed_shape
DType : uint16
Shape : (10,)
Number of Samples : 2
Partial Remote Data Refs : False
As you can see, everything is as it should be!
[30]:
co.close()
Making changes to introduce diverged histories¶
Let’s now go back to our testbranch
branch and make some changes there so we can see what happens when changes don’t follow a linear history.
[31]:
co = repo.checkout(write=True, branch='testbranch')
[32]:
co.columns
[32]:
Hangar Columns
Writeable : True
Number of Columns : 1
Column Names / Partial Remote References:
- dummy_column / False
[33]:
co.columns['dummy_column']
[33]:
Hangar FlatSampleWriter
Column Name : dummy_column
Writeable : True
Column Type : ndarray
Column Layout : flat
Schema Type : fixed_shape
DType : uint16
Shape : (10,)
Number of Samples : 1
Partial Remote Data Refs : False
We will start by mutating sample 0
in dummy_arrayset
to a different value
[34]:
old_arr = co['dummy_column', '0']
new_arr = old_arr + 50
new_arr
[34]:
array([50, 51, 52, 53, 54, 55, 56, 57, 58, 59], dtype=uint16)
[35]:
co['dummy_column', '0'] = new_arr
Let’s make a commit here, then add some metadata and make a new commit (all on the testbranch
branch).
[36]:
co.commit('mutated sample `0` of `dummy_column` to new value')
[36]:
'a=fcd82f86e39b19c3e5351dda063884b5d2fda67b'
[37]:
repo.log()
* a=fcd82f86e39b19c3e5351dda063884b5d2fda67b (testbranch) : mutated sample `0` of `dummy_column` to new value
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e : first commit with a single sample added to a dummy column
[38]:
co.metadata['hello'] = 'world'
[39]:
co.commit('added hellow world metadata')
[39]:
'a=69a08ca41ca1f5577fb0ffcf59d4d1585f614c4d'
[40]:
co.close()
Looking at our history how, we see that none of the original branches reference our first commit anymore.
[41]:
repo.log()
* a=69a08ca41ca1f5577fb0ffcf59d4d1585f614c4d (testbranch) : added hellow world metadata
* a=fcd82f86e39b19c3e5351dda063884b5d2fda67b : mutated sample `0` of `dummy_column` to new value
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e : first commit with a single sample added to a dummy column
We can check the history of the master
branch by specifying it as an argument to the log()
method.
[42]:
repo.log('master')
* a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94 (master) (new) : commit on `new` branch adding a sample to dummy_arrayset
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e : first commit with a single sample added to a dummy column
Merging (Part 2) Three Way Merge¶
If we now want to merge the changes on testbranch
into master
, we can’t just follow a simple linear history; the branches have diverged.
For this case, Hangar implements a Three Way Merge algorithm which does the following: - Find the most recent common ancestor commit
present in both the testbranch
and master
branches - Compute what changed between the common ancestor and each branch’s HEAD
commit - Check if any of the changes conflict with each other (more on this in a later tutorial) - If no conflicts are present, compute the results of the merge between the two sets of changes - Create a new commit
containing the merge results reference both branch HEAD
s as parents of the new commit
, and update the base
branch HEAD
to that new commit
’s commit_hash
[43]:
co = repo.checkout(write=True, branch='master')
Once again, as a user, the details are completely irrelevant, and the operation occurs from the same one-liner call we used before for the FF Merge.
[44]:
co.merge(message='merge of testbranch into master', dev_branch='testbranch')
Selected 3-Way Merge Strategy
[44]:
'a=002041fe8d8846b06f33842964904b627de55214'
If we now look at the log, we see that this has a much different look than before. The three way merge results in a history which references changes made in both diverged branches, and unifies them in a single commit
[45]:
repo.log()
* a=002041fe8d8846b06f33842964904b627de55214 (master) : merge of testbranch into master
|\
| * a=69a08ca41ca1f5577fb0ffcf59d4d1585f614c4d (testbranch) : added hellow world metadata
| * a=fcd82f86e39b19c3e5351dda063884b5d2fda67b : mutated sample `0` of `dummy_column` to new value
* | a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94 (new) : commit on `new` branch adding a sample to dummy_arrayset
|/
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e : first commit with a single sample added to a dummy column
Manually inspecting the merge result to verify it matches our expectations¶
dummy_arrayset
should contain two arrays, key 1
was set in the previous commit originally made in new
and merged into master
. Key 0
was mutated in testbranch
and unchanged in master
, so the update from testbranch
is kept.
There should be one metadata sample with they key hello
and the value "world"
.
[46]:
co.columns
[46]:
Hangar Columns
Writeable : True
Number of Columns : 1
Column Names / Partial Remote References:
- dummy_column / False
[47]:
co.columns['dummy_column']
[47]:
Hangar FlatSampleWriter
Column Name : dummy_column
Writeable : True
Column Type : ndarray
Column Layout : flat
Schema Type : fixed_shape
DType : uint16
Shape : (10,)
Number of Samples : 2
Partial Remote Data Refs : False
[49]:
co['dummy_column', ['0', '1']]
[49]:
[array([50, 51, 52, 53, 54, 55, 56, 57, 58, 59], dtype=uint16),
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype=uint16)]
[50]:
co.metadata
[50]:
Hangar Metadata
Writeable: True
Number of Keys: 1
[51]:
co.metadata['hello']
[51]:
'world'
The Merge was a success!
[52]:
co.close()
Conflicts¶
Now that we’ve seen merging in action, the next step is to talk about conflicts.
How Are Conflicts Detected?¶
Any merge conflicts can be identified and addressed ahead of running a merge
command by using the built in diff tools. When diffing commits, Hangar will provide a list of conflicts which it identifies. In general these fall into 4 categories:
- Additions in both branches which created new keys (samples / columns / metadata) with non-compatible values. For samples & metadata, the hash of the data is compared, for columns, the schema specification is checked for compatibility in a method custom to the internal workings of Hangar.
- Removal in
Master Commit/Branch
& Mutation inDev Commit / Branch
. Applies for samples, columns, and metadata identically. - Mutation in
Dev Commit/Branch
& Removal inMaster Commit / Branch
. Applies for samples, columns, and metadata identically. - Mutations on keys of both branches to non-compatible values. For samples & metadata, the hash of the data is compared; for columns, the schema specification is checked for compatibility in a method custom to the internal workings of Hangar.
Let’s make a merge conflict¶
To force a conflict, we are going to checkout the new
branch and set the metadata key hello
to the value foo conflict... BOO!
. Then if we try to merge this into the testbranch
branch (which set hello
to a value of world
) we see how hangar will identify the conflict and halt without making any changes.
Automated conflict resolution will be introduced in a future version of Hangar, for now it is up to the user to manually resolve conflicts by making any necessary changes in each branch before reattempting a merge operation.
[53]:
co = repo.checkout(write=True, branch='new')
[54]:
co.metadata['hello'] = 'foo conflict... BOO!'
[55]:
co.commit ('commit on new branch to hello metadata key so we can demonstrate a conflict')
[55]:
'a=95896880b33fc06a3c2359a03408f07c87bcc8c0'
[56]:
repo.log()
* a=95896880b33fc06a3c2359a03408f07c87bcc8c0 (new) : commit on new branch to hello metadata key so we can demonstrate a conflict
* a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94 : commit on `new` branch adding a sample to dummy_arrayset
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e : first commit with a single sample added to a dummy column
When we attempt the merge, an exception is thrown telling us there is a conflict!
[57]:
co.merge(message='this merge should not happen', dev_branch='testbranch')
Selected 3-Way Merge Strategy
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-57-1a98dce1852b> in <module>
----> 1 co.merge(message='this merge should not happen', dev_branch='testbranch')
~/projects/tensorwerk/hangar/hangar-py/src/hangar/checkout.py in merge(self, message, dev_branch)
1027 dev_branch=dev_branch,
1028 repo_path=self._repo_path,
-> 1029 writer_uuid=self._writer_lock)
1030
1031 for asetHandle in self._columns.values():
~/projects/tensorwerk/hangar/hangar-py/src/hangar/merger.py in select_merge_algorithm(message, branchenv, stageenv, refenv, stagehashenv, master_branch, dev_branch, repo_path, writer_uuid)
136
137 except ValueError as e:
--> 138 raise e from None
139
140 finally:
~/projects/tensorwerk/hangar/hangar-py/src/hangar/merger.py in select_merge_algorithm(message, branchenv, stageenv, refenv, stagehashenv, master_branch, dev_branch, repo_path, writer_uuid)
133 refenv=refenv,
134 stagehashenv=stagehashenv,
--> 135 repo_path=repo_path)
136
137 except ValueError as e:
~/projects/tensorwerk/hangar/hangar-py/src/hangar/merger.py in _three_way_merge(message, master_branch, masterHEAD, dev_branch, devHEAD, ancestorHEAD, branchenv, stageenv, refenv, stagehashenv, repo_path)
260 if conflict.conflict is True:
261 msg = f'HANGAR VALUE ERROR:: Merge ABORTED with conflict: {conflict}'
--> 262 raise ValueError(msg) from None
263
264 with mEnv.begin(write=True) as txn:
ValueError: HANGAR VALUE ERROR:: Merge ABORTED with conflict: Conflicts(t1=[(b'l:hello', b'2=d8fa6800caf496e637d965faac1a033e4636c2e6')], t21=[], t22=[], t3=[], conflict=True)
Checking for Conflicts¶
Alternatively, use the diff methods on a checkout to test for conflicts before attempting a merge.
It is possible to diff between a checkout object and:
- Another branch (diff.branch())
- A specified commit (diff.commit())
- Changes made in the staging area before a commit is made (diff.staged()) (for
write-enabled
checkouts only.)
Or via the CLI status tool between the staging area and any branch/commit (only a human readable summary is produced).
[58]:
merge_results, conflicts_found = co.diff.branch('testbranch')
[59]:
conflicts_found
[59]:
Conflicts(t1=Changes(schema={}, samples=(), metadata=(MetadataRecordKey(key='hello'),)), t21=Changes(schema={}, samples=(), metadata=()), t22=Changes(schema={}, samples=(), metadata=()), t3=Changes(schema={}, samples=(), metadata=()), conflict=True)
[60]:
conflicts_found.t1.metadata
[60]:
(MetadataRecordKey(key='hello'),)
The type codes for a Conflicts
namedtuple
such as the one we saw:
Conflicts(t1=('hello',), t21=(), t22=(), t3=(), conflict=True)
are as follow:
t1
: Addition of key in master AND dev with different values.t21
: Removed key in master, mutated value in dev.t22
: Removed key in dev, mutated value in master.t3
: Mutated key in both master AND dev to different values.conflict
: Bool indicating if any type of conflict is present.
To resolve, remove the conflict¶
[61]:
del co.metadata['hello']
# resolved conflict by removing hello key
co.commit('commit which removes conflicting metadata key')
[61]:
'a=e69ba8aeffc130c57d2ae0a8131c8ea59083cb62'
[62]:
co.merge(message='this merge succeeds as it no longer has a conflict', dev_branch='testbranch')
Selected 3-Way Merge Strategy
[62]:
'a=ef7ddf4a4a216315d929bd905e78866e3ad6e4fd'
We can verify that history looks as we would expect via the log!
[63]:
repo.log()
* a=ef7ddf4a4a216315d929bd905e78866e3ad6e4fd (new) : this merge succeeds as it no longer has a conflict
|\
* | a=e69ba8aeffc130c57d2ae0a8131c8ea59083cb62 : commit which removes conflicting metadata key
* | a=95896880b33fc06a3c2359a03408f07c87bcc8c0 : commit on new branch to hello metadata key so we can demonstrate a conflict
| * a=69a08ca41ca1f5577fb0ffcf59d4d1585f614c4d (testbranch) : added hellow world metadata
| * a=fcd82f86e39b19c3e5351dda063884b5d2fda67b : mutated sample `0` of `dummy_column` to new value
* | a=c1cf1bd6863ed0b95239d2c9e1a6c6cc65569e94 : commit on `new` branch adding a sample to dummy_arrayset
|/
* a=eaee002ed9c6e949c3657bd50e3949d6a459d50e : first commit with a single sample added to a dummy column