MEL Scripting Tutorial: Using The Tokenize Command

This MEL scripting tutorial will give you an in-depth look into how to use the tokenize command. Tokenize is used for taking a string and breaking it up into a number of parts so they may be used on their own. It's one of the most commonly used commands in the language and a strong knowledge of how to use it will help anyone write better procedures. This tutorial assumes a basic knowledge of running a MEL script in Maya.

Skill Level: Beginner

The tokenize MEL command is incredibly useful and very easy to use. You'll find once you know how it works and what it does, you'll end up using it in almost every script you write in one way or another. What tokenize does is essentially break a string up into different parts based on a splitter character, and return them into an empty string array. Why would you need to break up a string into different parts? There are many reasons you may need to do this. An example being let's say you're string is an Object.Attr and you need to parse out just the attribute, or perhaps just the object. You can use tokenize to split them up into their own strings. Maybe you have a path to a file and you want to tokenize it by slashes. Or, you have a comma-separated line of text and you want to split each field into an array. There are countless ways to use it and I think you'll find once you know how to use it, you'll use it constantly. Let's look at some examples:

"myObject.translateX" would tokenize into "myObject" and "translateX". Here's how we would do that:

// Here is our string, in the variable $myString
string $myString = "myObject.translateX" ;

// First we create an empty string array, to play our "parts" in.
string $buffer[] ;

// Now we'll tokenize our string and put the tokens into $buffer
tokenize $myString "." $buffer ;

// Notice the arguments:  string, splitter, array.

// Now if we print out $buffer we'll get:
print $buffer ;
// Result:
// myObject
// translateX

Note: Because we used the "." as the splitter, it is now gone from any return strings.

Besides just splitting a string up into different parts, tokenize will also return the number of parts your string was split up into. This can be useful for many reasons as well. Let's say you have a string that looks like this: "myObject,myAttribute,myValue,myMultiplier,myComment". This could be a comma-separated text file you're reading from, or data from a database table. So now for whatever reason let's say we want to know how many comma-separated fields are in this line of text. We can use tokenize for that as well.

// This is the original string
string $myLine = "myObject,myAttribute,myValue,myMultiplier,myComment" ;

// Tokenize it, but save the return of tokenize into an INT variable
string $buffer[] ;
int $numTokens = `tokenize $myLine "," $buffer` ;

print $numTokens ;
// Result: 5

print $buffer ;
// Result:
// myObject
// myAttribute
// myValue
// myMultiplier
// myComment

Now we have both our tokens saved into $buffer and the number of tokens saved into $numTokens. Now that you've got your 5 fields into your $buffer array, you can now define them properly.

// Define Vars
string $object = $buffer[0] ;
string $attribute = $buffer[1] ;
string $value = $buffer[2] ;
string $multiplier = $buffer[3] ;
string $comment = $buffer[4] ;

Note: Remember MEL arrays are "zero-based" so the first index of $buffer is going to be zero.

You may notice that I've defined $value and $multiplier as strings in the above example. Since they are coming from a comma-separated string and then tokenized into a string array they are technically strings. However, since they are most likely floats I can define them as such when I take them out of $buffer. You will get a Warning in Maya when you do this, but it won't stop your script. Same example but with redefined vars below:

// Define Vars, But Assign To Float
string $object = $buffer[0] ;
string $attribute = $buffer[1] ;
float $value = $buffer[2] ;
float $multiplier = $buffer[3] ;
string $comment = $buffer[4] ;

Now we've got five values that we took from our comma-separated text line, all in their own variables with no commas. Very useful! Now, what if we need to save this data out again? How do you form a line again with all these variables? The practical way would be to do this:

// Redefine CSV String
string $myNewLine = ($object+","+$attribute+","+$value+","+$multiplier+","+$comment) ;

This will put all your variables into a new, single string variable. But, if you've done this sort of thing you may have noticed that this will produce an error in MEL, because we're attempting to concatenate two different types of variables a string. In MEL, we can't combine floats and strings together into a string. (Thank you Python!) So, the alternative is to just use our original string array from your tokenize, along with the MEL command: stringArrayToString.

// Redefine CSV String
string $newLine = stringArrayToString($buffer, ",") ;

That will reassemble our string array into a single line again, and use the comma as the splitter. You can use anything you want as a splitter as long as it's a valid character.

Play around with tokenize a bit and see what you can come up with. It's one of the MEL commands I use the most and getting comfortable with how it works and how to use it will improve your scripting a lot!

Looking for more tutorials? Check out Script Swell's Technical Artist Tutorials page.

MEL: Remove Namespaces From A String or Object Name

Dealing with namespaces is a pain in the ass. Period.

Veteran TD's and Maya guys will tell you how easy it is and that there are so many ways to deal with them and this and that. They are correct. However, they are still a giant, giant pain in the ass.

Thankfully there's a simple trick to remove namespaces from using the tokenize MEL command.

What tokenize does is split a string into multiple parts using a character(s) as the splitting point. For example, if an object is named this_Object if you were to tokenize it and split it using the character "_" than you would end up with two tokens: "this" and "Object".

We know that if an object has a namespace, or in some cases multiple namespaces that the object name will always be prefixed with the namespace and a colon. (:)

So, let's tokenize it out using a colon.

string $objectName = "Character:r_arm_CTRL" ; // Our Object Name
string $buffer[] ; // Create Empty Array To Hold Tokens
tokenize $objectName ":" $buffer ; // Split $objectName by ":"

// Now...
string $namespace = $buffer[0] ; // "Character"
string $object = $buffer[1] ; // "r_arm_CTRL"

With this example we've done a basic namespace removal. However, this is prone to errors for multiple reasons. What if you have multiple namespaces? By hardcoding which index you're getting from $buffer you will end up potentially grabbing the wrong data. A way around this is to use a programming trick to always grab the very last item in an array. As with namespaces, we always know the last item will be the object itself so we'll end up with the correct string.

string $objectName = "Character:r_arm_CTRL" ;
string $buffer[] ;
tokenize $objectName ":" $buffer ;

string $objectWithoutNamespace = $buffer[size($buffer)-1] ;

By getting the size of the array, and subtracting one, you'll end up with the last index of a zero-based array. If it were Maxscript, arrays are one-based (they start with one, not zero) so you wouldn't need to subtract one from the size.

(If you're curious, the equivalent of tokenize in Maxscript is filterString)

So there you are, a quick, easy, non-dirty way of getting rid of namespaces in an object's name.