MEL Scripting Tutorial: Using The Tokenize Command

This MEL scripting tutorial will give you an in-depth look into how to use the tokenize command. Tokenize is used for taking a string and breaking it up into a number of parts so they may be used on their own. It's one of the most commonly used commands in the language and a strong knowledge of how to use it will help anyone write better procedures. This tutorial assumes a basic knowledge of running a MEL script in Maya.

Skill Level: Beginner

The tokenize MEL command is incredibly useful and very easy to use. You'll find once you know how it works and what it does, you'll end up using it in almost every script you write in one way or another. What tokenize does is essentially break a string up into different parts based on a splitter character, and return them into an empty string array. Why would you need to break up a string into different parts? There are many reasons you may need to do this. An example being let's say you're string is an Object.Attr and you need to parse out just the attribute, or perhaps just the object. You can use tokenize to split them up into their own strings. Maybe you have a path to a file and you want to tokenize it by slashes. Or, you have a comma-separated line of text and you want to split each field into an array. There are countless ways to use it and I think you'll find once you know how to use it, you'll use it constantly. Let's look at some examples:

"myObject.translateX" would tokenize into "myObject" and "translateX". Here's how we would do that:

// Here is our string, in the variable $myString
string $myString = "myObject.translateX" ;

// First we create an empty string array, to play our "parts" in.
string $buffer[] ;

// Now we'll tokenize our string and put the tokens into $buffer
tokenize $myString "." $buffer ;

// Notice the arguments:  string, splitter, array.

// Now if we print out $buffer we'll get:
print $buffer ;
// Result:
// myObject
// translateX

Note: Because we used the "." as the splitter, it is now gone from any return strings.

Besides just splitting a string up into different parts, tokenize will also return the number of parts your string was split up into. This can be useful for many reasons as well. Let's say you have a string that looks like this: "myObject,myAttribute,myValue,myMultiplier,myComment". This could be a comma-separated text file you're reading from, or data from a database table. So now for whatever reason let's say we want to know how many comma-separated fields are in this line of text. We can use tokenize for that as well.

// This is the original string
string $myLine = "myObject,myAttribute,myValue,myMultiplier,myComment" ;

// Tokenize it, but save the return of tokenize into an INT variable
string $buffer[] ;
int $numTokens = `tokenize $myLine "," $buffer` ;

print $numTokens ;
// Result: 5

print $buffer ;
// Result:
// myObject
// myAttribute
// myValue
// myMultiplier
// myComment

Now we have both our tokens saved into $buffer and the number of tokens saved into $numTokens. Now that you've got your 5 fields into your $buffer array, you can now define them properly.

// Define Vars
string $object = $buffer[0] ;
string $attribute = $buffer[1] ;
string $value = $buffer[2] ;
string $multiplier = $buffer[3] ;
string $comment = $buffer[4] ;

Note: Remember MEL arrays are "zero-based" so the first index of $buffer is going to be zero.

You may notice that I've defined $value and $multiplier as strings in the above example. Since they are coming from a comma-separated string and then tokenized into a string array they are technically strings. However, since they are most likely floats I can define them as such when I take them out of $buffer. You will get a Warning in Maya when you do this, but it won't stop your script. Same example but with redefined vars below:

// Define Vars, But Assign To Float
string $object = $buffer[0] ;
string $attribute = $buffer[1] ;
float $value = $buffer[2] ;
float $multiplier = $buffer[3] ;
string $comment = $buffer[4] ;

Now we've got five values that we took from our comma-separated text line, all in their own variables with no commas. Very useful! Now, what if we need to save this data out again? How do you form a line again with all these variables? The practical way would be to do this:

// Redefine CSV String
string $myNewLine = ($object+","+$attribute+","+$value+","+$multiplier+","+$comment) ;

This will put all your variables into a new, single string variable. But, if you've done this sort of thing you may have noticed that this will produce an error in MEL, because we're attempting to concatenate two different types of variables a string. In MEL, we can't combine floats and strings together into a string. (Thank you Python!) So, the alternative is to just use our original string array from your tokenize, along with the MEL command: stringArrayToString.

// Redefine CSV String
string $newLine = stringArrayToString($buffer, ",") ;

That will reassemble our string array into a single line again, and use the comma as the splitter. You can use anything you want as a splitter as long as it's a valid character.

Play around with tokenize a bit and see what you can come up with. It's one of the MEL commands I use the most and getting comfortable with how it works and how to use it will improve your scripting a lot!

Looking for more tutorials? Check out Script Swell's Technical Artist Tutorials page.