Move megatron conversion script and add rope arguments

[loubnabnl]

This copies megatron conversion script to gpt_bigcode folder, adds RoPE arguments to GPTBigcodeConfig + a test for verifying generatio

[RaymondLi0]

Left a comment on the scale
Also Megatron's implementation of Rope looks a bit different, I'm having a closer look

[RaymondLi0]

Does it work with the scale change?

Otherwise, I'm not certain, but Megatron's rope implementation seems to process the tensor differently.
We may need to change this line https://github.com/bigcode-project/transformers/blob/sc2-ablations/src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py#L92
to:
complex_tensor = torch.view_as_complex(tensor.float().view(*tensor.shape[:-1], -1, 2, rope_frequencies.size(-1)).transpose(-2, -1))

[loubnabnl]

The change seems to work, the model generates this, thanks! I'll see what HumanEval numbers I get and if the change you mentioned is needed

def fibonnaci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonnaci(n-1) + fibonnaci(n-1)

[RaymondLi0]

Ah ok then, perfect!

[loubnabnl]

I spoke too early, it seems that the modeling worked for fibonacci example but not for others, below are some examples I got

Testing generation with prompt 'def separate_paren_groups(paren_string: str) -> List[str]:
    """ Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
    separate those group into separate strings and return the list of those.
    Separate groups are balanced (each open brace is properly closed) and not nested within each other
    Ignore any spaces in the input string.
    >>> separate_paren_groups('( ) (( )) (( )( ))')
    ['()', '(())', '(()())']
    """'
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
def separate_paren_groups(paren_string: str) -> List[str]:
    """ Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
    separate those group into separate strings and return the list of those.
    Separate groups are balanced (each open brace is properly closed) and not nested within each other
    Ignore any spaces in the input string.
    >>> separate_paren_groups('( ) (( )) (( )( ))')
    ['()', '(())', '(()())']
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
    """
Testing generation with prompt 'def fibonnaci(n'
Input ids: tensor([[  588, 28297,   264,    93, 17555,    23,    93]], device='cuda:0')
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
def fibonnaci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonnaci(n-1) + fibonnaci(n-1)

def fibonnaci(n):
        return fibonnaci(n)

But the change you suggested seems to fix the issue although HumanEval score doesn't seem very high which could also be due to something else (will post details on slack).

Btw I added .contiguous() to the code you suggested to avoid the Tensor must have a last dimension with stride 1 error)

generations after the change

def separate_paren_groups(paren_string: str) -> List[str]:
    """ Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
    separate those group into separate strings and return the list of those.
    Separate groups are balanced (each open brace is properly closed) and not nested within each other
    Ignore any spaces in the input string.
    >>> separate_paren_groups('( ) (( )) (( )( ))')
    ['()', '(())', '(()())']
    """
    # TODO: Implement this function
    return [x for x in paren_string.split('(') if x!= '']


def main():
    # Test cases
    assert separate_paren_groups('( ) (( )) (( )( ))') == ['()', '(())', '(()())']
    assert separate_paren_groups('( ) (( )) (( )( ))') == ['()', '(())', '(()())']
    assert separate_paren_groups('( ) (( )) (( )( ))') == ['()', '(())', '(()())']
    assert separate_paren_groups('( ) (( )) ((
Testing generation with prompt 'def fibonnaci(n'
Input ids: tensor([[  588, 28297,   264,    93, 17555,    23,    93]], device='cuda:0')
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
def fibonnaci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonnaci(n-1) + fibonnaci(n-2)

print(fibonnaci(10))

# +
# 10.1

def fibonnaci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonnaci(n-1) + fibonnaci(n-2)