561698 - Repo restruct: Migrate repositories - Add migration scripts
Change-Id: I388f049f7b0282c2f349457ed670ac6d86fbe093
Signed-off-by: Simon Skoczylas <simon.skoczylas@karakun.com>
diff --git a/tools/migration/README.md b/tools/migration/README.md
new file mode 100644
index 0000000..8dd7cca
--- /dev/null
+++ b/tools/migration/README.md
@@ -0,0 +1,47 @@
+## Short manual for the restructure.sh BASH script.
+
+The restructure.sh script is intended for use in the Eclipse openMDM Working Group to merge multiple Git repositories into one.
+See: https://bugs.eclipse.org/bugs/show_bug.cgi?id=561698
+
+### First of all, change the variables
+Change the variables inside of restructure.sh if the current values don't fit your needs
+
+* COMMIT_USER for the Git user
+* COMMIT_MAIL for the Git email address
+* COMMIT_PREFIX for the Prefix used in the merge commit messages
+* REMOTE_NAME the name of the git remote for the new repository.
+* SUBDIR the sub directory which will be used to clone the necessary git repositories
+* NEW_REPO_NAME the name of the new git repository which will only be created on your local maschine
+* NEW_REPO_DIR the name of the new git bare repository which will only be created on your local maschine (should end with .git)
+* BASE, DEFAULT, ODSADAPTER and NUCLEUS as the used Git repositories
+* REPOS as the list of the current Git repositories
+
+### How does the script work?
+The script will create a new sub directory (variable SUBDIR) on do all the work inside that directory.
+
+The script creates two local Git repositories, a bare one (will be used as remote) and a normal one (will be used as working copy).
+
+After that, the script will add all defined repositories (variable REPOS) as new remotes to the working copy.
+For each added remote the script will detect the exising branches and create temporary mirror branches in the new working copy.
+
+Then every project will be merged into the working copy (with flag --allow-unrelated-histories) and moved into an own project directory. This step will be repeated for every branch detected.
+
+If a branch does not exits in a project, the master branch will be used to create that project directory on that branch.
+
+> Tags must be "recreated" by hand.
+
+### Possible handling of Tags
+And the end of the script, all Tags found in the Git repositories will be listed with the corresponding commit hashes.
+
+To create the correct state of a previous Tag take this steps:
+
+1. Go to the new wokring copy directory (e.g. restructuring/org.eclipse.mdm)
+2. Checkout the clean branch init/empty
+3. Create a new branch to work on the Tag (e.g. tag-work/5.1.0M8)
+4. Switch to the branch
+5. Merge the first tag form a project by hash (e.g merge --allow-unrelated-histories -m "Some message" 7545d24dae04b733cf34843adaa8754941952e8a)
+6. Create a new subfolder with the name of the project (e.g. org.eclipse.mdm.nucleus)
+7. Move everything besides the .git and the new subfolder directory into that directory (use git move to keep Git history)
+8. Commit the change to that branch and repeat from step 5. for the same Tag from the other projects
+
+At the end you can create a new Tag on the current branch (e.g. tag-work/5.1.0M8) and switch back to the init/empty branch (maybe you have to use git reset --hard to get the clean branch back) and repeat the steps for another Tag
\ No newline at end of file
diff --git a/tools/migration/restructure.sh b/tools/migration/restructure.sh
new file mode 100755
index 0000000..0db4828
--- /dev/null
+++ b/tools/migration/restructure.sh
@@ -0,0 +1,306 @@
+#!/bin/bash
+
+#/********************************************************************************
+# * Copyright (c) 2015-2019 Contributors to the Eclipse Foundation
+# *
+# * See the NOTICE file(s) distributed with this work for additional
+# * information regarding copyright ownership.
+# *
+# * This program and the accompanying materials are made available under the
+# * terms of the Eclipse Public License v. 2.0 which is available at
+# * http://www.eclipse.org/legal/epl-2.0.
+# *
+# * SPDX-License-Identifier: EPL-2.0
+# *
+# ********************************************************************************/
+
+# This script is related to the openMDM requirement 560684 - https://bugs.eclipse.org/bugs/show_bug.cgi?id=560684
+
+# Interesting articles about merging different git repositories into one
+# https://medium.com/altcampus/how-to-merge-two-or-multiple-git-repositories-into-one-9f8a5209913f
+# https://gist.github.com/msrose/2feacb303035d11d2d05
+
+
+##### Configuration begin
+
+# COMMIT_USER, COMMIT_MAIL will be used as User and Mail adress for the commits (merge) in the new repository
+# COMMIT_PREFIX will be used as part of the commit message Matthias Koller <m.koller@peak-solution.de>
+# Please change COMMIT_USER, COMMIT_MAIL and COMMIT_PREFIX if necessary
+COMMIT_USER="Simon Skoczylas"
+COMMIT_MAIL="simon.skoczylas@karakun.com"
+COMMIT_PREFIX="561698 - "
+
+# The name of the git remote for the new repository.
+REMOTE_NAME=restructuring
+# The sub directory which will be used to clone the necessary git repositories
+SUBDIR=restructuring
+# The name of the new git repository which will only be created on your local maschine
+NEW_REPO_NAME="org.eclipse.mdm"
+# The name of the new git bare repository which will only be created on your local maschine (should end with .git)
+NEW_REPO_DIR="$NEW_REPO_NAME.git"
+
+# Items for projects with name and location.
+BASE=("org.eclipse.mdm.api.base" "git://git.eclipse.org/gitroot/mdmbl/org.eclipse.mdm.api.base.git")
+DEFAULT=("org.eclipse.mdm.api.default" "git://git.eclipse.org/gitroot/mdmbl/org.eclipse.mdm.api.default.git")
+ODSADAPTER=("org.eclipse.mdm.api.odsadapter" "git://git.eclipse.org/gitroot/mdmbl/org.eclipse.mdm.api.odsadapter.git")
+NUCLEUS=("org.eclipse.mdm.nucleus" "git://git.eclipse.org/gitroot/mdmbl/org.eclipse.mdm.nucleus.git")
+
+# List of the defined items
+REPOS=(BASE DEFAULT ODSADAPTER NUCLEUS)
+
+##### Configuration end!
+
+echo "WELCOME! This script helps to merge the existing openMDM git repositories into one!"
+echo "At the end you must add an git remote the the new created local repository and push it to the remote if everything is fine"
+
+echo
+
+echo "Will work with the following configuration:"
+echo "-- Commiter name: '${COMMIT_USER}'"
+echo "-- Commiter email adress: '${COMMIT_MAIL}'"
+echo "-- Commit message prefix: '${COMMIT_PREFIX}'"
+
+echo
+
+# Output the list of repositories which will be cloned
+echo "Will work with the following repositories:"
+for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+ echo "-- ${list[0]} from ${list[1]}"
+done
+
+# Create the new directory where all the work will be done. If the directory already exists, it will be DELETED!
+if [ -d "$SUBDIR" ]; then
+ echo
+ echo "WARNING: The directory '$SUBDIR' already exists and will be DELETED!"
+ echo
+ rm -rf $SUBDIR/
+ mkdir $SUBDIR
+else
+ mkdir $SUBDIR
+fi
+cd $SUBDIR
+
+read -p "Press Enter to continue"
+
+echo "Working inside:"
+pwd
+
+echo
+
+# Create a new directory for the git bare repository
+REMOTE_DIR=$(pwd)/$NEW_REPO_DIR
+echo "Remote directory is $REMOTE_DIR/"
+if [ -d "$NEW_REPO_DIR" ]; then
+ echo "$NEW_REPO_DIR already exists"
+else
+ mkdir $NEW_REPO_DIR
+fi
+
+# Initialize the new git bare repository
+cd $NEW_REPO_DIR
+if [ -d ".git" ]; then
+ echo
+ echo "WARNING: Already a git repository, this should not happen!"
+ echo
+ exit 1
+else
+ git init --bare
+fi
+cd ..
+
+# Clone the new repository to create a working copy where all the magic will happen!
+git clone file://$REMOTE_DIR $NEW_REPO_NAME
+cd $NEW_REPO_NAME
+
+# Set the user and mail for the working repository
+git config user.name "\"$COMMIT_USER\""
+git config user.email "\"$COMMIT_MAIL\""
+
+# Create a list with just the names of the projects
+REPO_NAMES=()
+for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+ REPO_NAMES+=(${list[0]})
+done
+
+echo
+
+# Create a dummy commit+cleanup in the new repository to be able to merge into this repository
+echo "Create initial commit"
+touch initial_commit.md
+git add initial_commit.md
+git commit -s -q -m "\"$COMMIT_PREFIX Initial commit in the new repository\""
+rm initial_commit.md
+git add initial_commit.md
+git commit -s -q -m "\"$COMMIT_PREFIX Cleanup the new repository\""
+git checkout -b init/empty
+git push -q --set-upstream origin init/empty
+git checkout master
+git push -q --set-upstream origin master
+
+echo
+
+# Add the existing projects as remotes to the working repository
+echo "Add remotes"
+for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+ echo "Add remote '${list[0]}' with '${list[1]}'"
+ git remote add ${list[0]} ${list[1]}
+done
+
+# Fetch data from remote without merging (don't pull here)
+echo "Fetch from all"
+git fetch -q --all --no-tags
+
+echo
+
+# Create all branches which were found in the projects
+echo "Create branches"
+for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+ BRANCHES=($(git branch -a --list "*${list[0]}*"))
+ echo "Branches for '${list[0]}'"
+ for branch in ${BRANCHES[*]}; do
+ BRANCH_REGEX="remotes/${list[0]}/"
+ CURRENT_BRANCH="${branch/$BRANCH_REGEX/}"
+ if [ $CURRENT_BRANCH != "master" ]; then
+ echo "Create branch '$CURRENT_BRANCH'"
+ git branch $CURRENT_BRANCH
+ fi
+ done
+done
+
+echo
+
+# The magic will happen here!
+# 1. Checkout every branch of every project into a temporary branch
+# 2. Move (with git mv) all files into a separate directory which is named like the project
+# 3. Checkout the relevant branch (which was created before)
+# 4. Merge the temporary branch into the new branch and allow to merge unrelated histories
+for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+
+ RECENT_COMMIT=''
+ BRANCHES=($(git branch -a --list "*${list[0]}*"))
+ for branch in ${BRANCHES[*]}; do
+ BRANCH_REGEX="remotes/${list[0]}/"
+ CURRENT_BRANCH="${branch/$BRANCH_REGEX/}"
+ echo "Working on branch '${list[0]}/$CURRENT_BRANCH'"
+
+ echo "Checkout '${list[0]}/$CURRENT_BRANCH' as 'temp/${list[0]}/$CURRENT_BRANCH'"
+ git checkout -q -b temp/${list[0]}/$CURRENT_BRANCH ${list[0]}/$CURRENT_BRANCH
+
+ echo "Collect files"
+ shopt -s dotglob
+ FILE_LIST=()
+ for entry in "."/*; do
+ #echo "Entry is $entry"
+ if [ $entry != "./.git" ]; then
+ ADD=true
+ for r in ${REPO_NAMES[*]}; do
+ if [ $entry = "./$r" ]; then
+ #echo "Omit repo $r"
+ ADD=false
+ break
+ fi
+ done
+ if [ $ADD = true ]; then
+ #echo "Add $entry"
+ FILE_LIST+=($entry)
+ fi
+ fi
+ done
+ shopt -u dotglob
+
+ echo "Move files for ${list[0]}/$CURRENT_BRANCH"
+ if [ ! -d ${list[0]} ]; then
+ echo "Creating folder for project ${list[0]}"
+ mkdir ${list[0]}
+ fi
+
+ for file in ${FILE_LIST[*]}; do
+ git mv $file ${list[0]}/$file
+ done
+ git commit -s -q -m "\"$COMMIT_PREFIX Move files into separate folder for '${list[0]}/$CURRENT_BRANCH'\""
+
+ echo "Back to $CURRENT_BRANCH"
+ git checkout -q $CURRENT_BRANCH
+
+ echo "Merge 'temp/${list[0]}/$CURRENT_BRANCH' into '$CURRENT_BRANCH'"
+ git merge --signoff -q --allow-unrelated-histories -m "\"$COMMIT_PREFIX Merge '${list[0]}/$CURRENT_BRANCH' into '$CURRENT_BRANCH'\"" temp/${list[0]}/$CURRENT_BRANCH
+
+ echo
+ done
+done
+
+# Get back to master
+git checkout -q master
+
+# Check wether all projects directories exits on every branch. Merge temporary master if the project directories does not exits on that branch
+echo "Check branches"
+BRANCHES=($(git --no-pager branch --list | sed -e "s/* master//" | sed -e "s/temp.*//"))
+for branch in ${BRANCHES[*]}; do
+ if [ $branch != "master" ] && [ $branch != "init/empty" ]; then
+ git checkout -q $branch
+ for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+ if [ ! -d ${list[0]} ]; then
+ echo "Must create ${list[0]} on $branch"
+ git merge --signoff -q --allow-unrelated-histories -m "\"$COMMIT_PREFIX Merge '${list[0]}/master' into '$branch'\"" temp/${list[0]}/master
+ fi
+ done
+ fi
+done
+
+# Get back to master
+git checkout -q master
+
+echo
+
+# Remove all temporary branches
+echo "Clean up"
+BRANCHES=($(git --no-pager branch --list "temp*"))
+for branch in ${BRANCHES[*]}; do
+ echo "Delete temporary branch $branch"
+ git branch -q -D $branch
+done
+
+echo
+
+# Remove the remotes from the working repository
+echo "Remove remotes"
+for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+ echo "Remove remote '${list[0]}' with '${list[1]}'"
+ git remote remove ${list[0]}
+done
+
+cd ..
+
+echo
+pwd
+
+echo "Clone repositories to list available tags"
+for repo in ${REPOS[*]}; do
+ temp="$repo[@]"
+ list=("${!temp}")
+
+ echo "Will now clone '${list[0]}' from '${list[1]}'"
+ git clone -q ${list[1]} ${list[0]}
+ cd ${list[0]}
+ TAGS=($(git --no-pager tag --list))
+ for tag in ${TAGS[*]}; do
+ COMMIT_HASH=$(git rev-list -n 1 ${tag})
+ echo "Found tag ${tag} in ${list[0]} as ${COMMIT_HASH}"
+ done
+ cd ..
+ rm -rf ${list[0]}/
+done
\ No newline at end of file
diff --git a/tools/migration/tags.sh b/tools/migration/tags.sh
new file mode 100755
index 0000000..c74145f
--- /dev/null
+++ b/tools/migration/tags.sh
@@ -0,0 +1,102 @@
+#!/bin/bash
+
+#/********************************************************************************
+# * Copyright (c) 2015-2019 Contributors to the Eclipse Foundation
+# *
+# * See the NOTICE file(s) distributed with this work for additional
+# * information regarding copyright ownership.
+# *
+# * This program and the accompanying materials are made available under the
+# * terms of the Eclipse Public License v. 2.0 which is available at
+# * http://www.eclipse.org/legal/epl-2.0.
+# *
+# * SPDX-License-Identifier: EPL-2.0
+# *
+# ********************************************************************************/
+
+# This script is related to the openMDM requirement 560684 - https://bugs.eclipse.org/bugs/show_bug.cgi?id=560684
+
+
+
+##### Configuration begin
+
+# COMMIT_USER, COMMIT_MAIL will be used as User and Mail adress for the commits (merge) in the new repository
+# COMMIT_PREFIX will be used as part of the commit message
+# Please change COMMIT_USER, COMMIT_MAIL and COMMIT_PREFIX if necessary
+COMMIT_USER="Simon Skoczylas"
+COMMIT_MAIL="simon.skoczylas@karakun.com"
+COMMIT_PREFIX="561698 - "
+
+# The name of the git remote for the new repository.
+REMOTE_NAME=restructuring
+# The sub directory which will be used to clone the necessary git repositories
+SUBDIR=restructuring
+# The name of the new git repository which will only be created on your local maschine
+NEW_REPO_NAME="org.eclipse.mdm"
+# The name of the new git bare repository which will only be created on your local maschine (should end with .git)
+NEW_REPO_DIR="$NEW_REPO_NAME.git"
+
+PROJECT_ORDER=("org.eclipse.mdm.api.base" "org.eclipse.mdm.api.default" "org.eclipse.mdm.api.odsadapter" "org.eclipse.mdm.nucleus")
+TAG_520M1=("5.2.0M1" "63439c6c25d38a7ef59e49175b0268ad130b2a85" "b667b1a5db701d6d136a2eae67027e9d234aa4b6" "6caabd454c817022de33df705c873b14906cd11b" "c7163c74e59bde6e27194cde183f1e7440b61960")
+TAG_510=("5.1.0" "4ac0ec4f7269b20f4c4a71cb190b8d3e18857adb" "172df8bf70a2060cf79b6736553e3344aeae3074" "6bb90c7829ee8d07dcca76e2102e3ff568531fa1" "80f4e05ebe1a229e46801a9d05f29b7ec9fe6ad8")
+TAG_500=("5.0.0" "9294001445e38f6b63f1ba0d853fdd555a65d832" "e9c319b1917d7d13919375342f646a04129f2dc4" "b1ea88258827bf4fe72b102965dcb9b2fb280b98" "b2c3eec059904212b85385dcc8030ed5b35231a0")
+
+TAGS=(TAG_520M1 TAG_510 TAG_500)
+
+##### Configuration end!
+
+echo "Will work with the following tags:"
+for tag in ${TAGS[*]}; do
+ temp="$tag[@]"
+ list=("${!temp}")
+ echo "Tag ${list[0]}"
+done
+
+cd $SUBDIR
+
+read -p "Press Enter to continue"
+
+cd $NEW_REPO_NAME
+# Set the user and mail for the working repository
+git config user.name "\"$COMMIT_USER\""
+git config user.email "\"$COMMIT_MAIL\""
+
+echo "Working inside:"
+pwd
+
+shopt -s extglob
+
+for tag in ${TAGS[*]}; do
+ temp="$tag[@]"
+ list=("${!temp}")
+ echo "Working on tag ${list[0]}"
+ git checkout init/empty
+ git checkout -b taggy/${list[0]}
+ omit=".git"
+ echo
+ i=0
+ for project in ${PROJECT_ORDER[*]}; do
+ i=$((i+1))
+ echo "Merge ${list[0]} for ${project} with ${list[i]}"
+ # git merge --allow-unrelated-histories -m "Get 5.2.0M1 for org.eclipse.mdm.api.default" b667b1a5db701d6d136a2eae67027e9d234aa4b6
+ git merge --signoff -q --allow-unrelated-histories -m "Get ${list[0]} for ${project}" ${list[i]}
+ mkdir ${project}
+ # git mv !(.git/|org.eclipse.mdm.api.base) org.eclipse.mdm.api.base/
+ omit="${omit}|${project}"
+ echo "Move ${project} to separate directory"
+ git mv !(${omit}) ${project}/
+ git mv .gitignore ${project}/
+ git commit -s -q -m "Move ${project} to separate directory"
+ done
+ echo "Create tag ${list[0]}"
+ git tag ${list[0]}
+ git checkout master
+ #git reset
+ #git clean -df
+ git branch -D taggy/${list[0]}
+ echo
+done
+git checkout master
+git branch -D init/empty
+
+shopt -u extglob
\ No newline at end of file
