Link external data (.csv) file

[Filip Vandueren]

Don’t add the .csv to your timeline.
Just import it in the project and reference it directly within your expressions. Use footage()
It’s the adding it to the timeline that takes ages, because after effects needs to create an interface to the data

I’ve found that manually looking through the csv’s sourceText with string-functions is also a lot faster than using the sourceData methods.

If you post a small excerpt of a few lines of csv I could help in more detail

Or see this thread:

https://forums.creativecow.net/thread/227/45348#45397

[Christine Tang]

Here’s an excerpt of my data:

My columns show, OBJECTID, POS X, POS Y, Rotational Angle, per object I am animating through the .csv. There’s roughly 50 objects with 3 values each. OBJECTID is a time value per frame.

How do I then change my code I posted earlier:

myData = thisComp.layer(“CompiledTrainsReduced2.csv”)(“Data”)(“Outline”);
i= timeToFrames(time);
if( i > 0 && i < myData(“OBJECTID”).numProperties-1)
{
initTime = myData(“OBJECTID”)(“OBJECTID ” + (i-1)).value;
endTime = myData(“OBJECTID”)(“OBJECTID ” + i).value;
initPosition = eval(myData(“ML1_X”)(“ML1_X “+(i-1)).value);
endPosition = eval(myData(“ML1_X”)(“ML1_X “+(i)).value);
linear(time,initTime,endTime,initPosition,endPosition)/7.9
}

ML1_X is my data set’s position X. And this expression is plugged into my layer, position x.

Using your footage code, I am trying to grab the data value from POS X to link to Position X in my layer and have it play by frame by frame by OBJECTID (which is time in my document).

[Filip Vandueren]

Can you copy the entire text of 1 or two rows (all columns)

Is this data that will change often and need to be re-imported ?
In other words, is it important that the data stays ‘live’ -> the CSV might change ?

If not, then it’s very possible that writing a script to create keyframes is a better solution.

[Christine Tang]

Data needs to stay live as it would constantly be changed/edited unfortunately..

[Filip Vandueren]

Ok,

No problem.
Like I said, the fastest way is to just look at the sourceText of the csv footage,
Make substring of the line you need (with a regexp that matches a unique ID)
You can then .split(“,”) to get an array.
Then you can just get the items out of the array as column indexes.

If there’s data for every single frame, then there shouldn’t be any linear() interpolation though?

If you paste a full header and line of data from the csv I can go in more concrete detail if you wish

[Christine Tang]

Thank you, please go into detail as I had help coming to my current expression…

14222_trains000.csv.zip

Here is the file. If I had help starting it off for the X axis I can continue and adapt the expression to my Y and Rotational axis. I use POINT_X_0 and POINT_Y_0 for one train and POINT_X_1 and POINT_Y_1 another train. In the example earlier “ML1_X” is the same as “POINT_X_0” in my document, I just renamed it for my clarity.

My goal is to plot pos X and pos Y from my data set to animate a Train moving by each frame. You mentioned the linear() might not needed, if there are better ways to approach this and make after effects less laggy, please let me know.

Apologies for my lack of expression knowledge. I am new to learning expressions. I took a look at your thread you linked and tried out footage() but did not figure out how to adapt it to work like my original expression did.

[Filip Vandueren]

OK, thanks for sharing.

That’s about 50 minutes worth of animation data…

The good thing: there is a row for every frame, so we don’t even have to search through the data to find a matching framenumber “f”,
we can just take the f’th line (element f of the array of all the text split in \r’s = returns ),


f=86613 + timeToFrames(time);
footage("14222_trains000.csv").sourceText.split("\r")[f];

(Nothing happens before frame 86613, so I’ve added that offset to my example.)

add the expression above to a text-Layer. name that text-layer “Row”.
you can view the data changing every frame.
Now, we will re-use the values in text-layer in other expressions in your train layers(rotation, position,…) because that’s a lot faster than starting every expression with the same lookup in the csv.

for example, point 0’s position would be:

row = eval("["+thisComp.layer("Row").text.sourceText+"]");
[row[6],row[7]] / 7.9;

(because x and y are columns 7 and 8 in the csv)

point 0’s rotation would be:

row = eval("["+thisComp.layer("Row").text.sourceText+"]");
row[5];

(because rotation is the 6th column)

I’ve done some tests:
Using After Effects’ built-in dataValue() methods can render a bit faster (after a half minute apparant system-freeze), but they are muuuuuch slower to work with in the timeline.
It feels as if the whole .csv has to be read and parsed before any manipulation (or rendering), but once that’s “cached” it’s ok.
But, If you for example disable an expression and enable it again, then DataValue() would think again for 30 seconds.

My sourceText-method just looks up the needed values on the fly.
If the outside file changes or needs to be re-imported, that also takes about 30 seconds on my Macbook.

I was able to render the visualization of the final 1minute 47 seconds of data (the only part where something happens) in under 2 minutes .

project and testrender:
https://we.tl/t-pwpd5YqeKi

[Filip Vandueren]

A last tip, but potentially Important:

When you import a .csv file,
– in the File Dialog enable “All Files”
– select the .csv file, but beofre importing it;
– Change the file import Format from “Comma Separated Value” to “Javascript”

Boom: loads instantly, no half minute waiting for csv parsing.
If you accidentally drop it in a comp: no 10 hour waiting, just a “useless” layer that gets added to your comp.

Javascript files behave as an inert chunk of text, just waiting to be interpreted or read by an expression, unlike a .csv or .tsv (.txt) that wants to parse itself to create handlers to make its data accessible.

Now, to be clear: You cannot use dataValue() methods on it at all.
Only manipulate the footage().sourceText.

But as I’ve shown – for now as of CC2020- for large datasets it is much much faster than using After Effects’ own csv parsing, no crashing, no waiting.

[Christine Tang]

Thank you Filip,

I’ve spent the last day working with your file and it works extremely well! Your explanations make it easy to understand with my little knowledge of expressions, thank you.

I ran into the same problem you mentioned with dataValue(), it renders faster but in that it slows after effects down extremely. Your work around is much better!

This final tip I will explore today as that was something I was struggling with. My .csv’s take hours to import and when I close my after effects and re-open my file it takes hours to open as well.

I’ll edit this text blurb to let you know if all is successful but once again I appreciate your time guiding me through this! Thank you so much.

[Filip Vandueren]

Does even my example file take hours to load?
As I mentioned, it opens in about 30 seconds for me (before I added the”JavaScript file” hack)