Creative Communities of the World Forums

The peer to peer support community for media production professionals.

Forums Adobe After Effects Expressions Parsing JSON Files – Fastest possible way?

  • Parsing JSON Files – Fastest possible way?

  • Tomas Bumbulevičius

    September 1, 2021 at 1:17 pm

    Hello guys,

    I have been dealing with this issue for a while, and found similar topics online. Seems quite logical – the bigger the json, the longer parsing time. But the question is – how long is FAIR?

    I am using external json2 library to parse JSON’s and its slow.

    In Extendscript, it takes ages to parse the JSON into memory. It seems that speed falls exponentially – the bigger the file. And the bad thing is – I am not talking something extraordinary huge – 1MB file can take minutes.

    Is there a way? I saw this thread about exactly the same thing, which was left for dead:
    https://creativecow.net/forums/thread/scripting-best-way-to-parse-a-large-json-fileae/

    Found very interesting presentation about overall parsing of JSON’s, if someone wants to look at in depth:
    https://www.infoq.com/presentations/simdjson-parser/

    Do you have any ideas on how to get around this issue and speed up the parsing of ANY json file, without affecting the structure itself?

    Any insights would be much appreciated.

    Thank you.

    Scripting – Best way to parse a large JSON file?

  • Andrei Popa

    September 2, 2021 at 8:56 am

    I remember I ran some tests for json creation and parsing with json2. In the end it looked like creating file with .toSource() and evaluating it with eval() was ages faster. And I think that if you do not have functions in your variable, the result from .toSource() can be used as json syntax.

  • Tomas Bumbulevičius

    September 2, 2021 at 10:41 am

    Hey Andrei, thanks a lot for your insight! As I am not aware of this method, could you please elaborate further on this, whether:
    a) JSON file structure itself needs to be appended with toSource()?
    b) JSON file should be read as string and then appended and evaled?
    c) Anything else?

    Thanks, A!

  • Andrei Popa

    September 3, 2021 at 3:15 pm

    I tried to respond to you yesterday but for some reason it was considered spam and I couldn’t.

    The simplest way I can put it is this:

    toSource() is equivalent to JSON.stringify(). Except that it adds two parentheses, one at the start and one at the end. Better yet than JSON.stringify, toSource() can also serialize functions contained in your objects. The result is a string.

    eval() works like JSON.parse(). I used it several times with json files and it worked well. Way faster than JSON.parse(). Also, eval() is capable of evaluation objects that contain functions. The parameter sent to eval() must be a string.


    some basic example:

    var a = {some complicated object};

    var aString = a.toSource();


    You can save aString in some file.

    if you want to use var a again:

    var a = eval(aString);

    I didn’t find a lot against these 2 functions, except that you should not use eval() because it could cause harm by evaluating something someone else could pass to it. But I think it is safe if you use it in AE scripts.


  • Tomas Bumbulevičius

    September 6, 2021 at 7:56 am

    Hey Andrei, thanks a lot for your response. I am aware of eval’s safety issues, that it can evaluate malware functions added as objects.

    On a different note, what you say makes total sense when data is already available on hand. But what about when its read from file? Should it be opened and scanned line-by-line intro string, to be able to source it? Or what would be approach in such external file case?

    I will test it myself, but curious what do you think. Thanks as always!

  • Filip Vandueren

    September 6, 2021 at 1:25 pm

    You can add an external file as a source in the project, when you explicitly import it as a javascript file, the import is instantaneous, but when you for example import a giant .csv file as a txt-file, After effects can take a while parsing it on import, even before you do anything.

    Anyway, you can should be able to simply use eval(footage(“myText.txt”).sourceText); once it’s in your project

  • Andrei Popa

    September 6, 2021 at 3:38 pm

    I use some json structure for some of my scripts. They go to as much as 1.4 mb and the load is almost instant. As method I open the file, read the whole content inside a variable and then eval that string variable. Then I use what part of the new created object I need.

    The files were created with toSource() from a big var object.

  • Tomas Bumbulevičius

    September 11, 2021 at 7:32 am

    Hey Andrei, thanks a lot for your in-depth responses. I tried it in multiple ways but received same results, thus will break it down real quick in case I miss something.

    1. I have .json file on a disk. I read it line-by-line as a plain string into memory, into variable A.
    2. Then turn variable A into variable B as A.toSource(). It makes whole text into a an object of “New String” – I guess thats what toSource is expected to do.
    3. Presumably, evaluating variable B into variable C should get an object from strings, right? as var C = eval(B). What it actually returns is back to the same A string, without making it an object.

    Now I found different topics where extra brackets are suggested as eval(‘(‘ + B + ‘)’) is executed, but it didn’t made a difference.

    I made the same approach from file as well as object written as string to variable (to make sure there is no read from file issues) but result remained the same – toSource string eval’ed became the same string.

    It might be that I am not understanding the concept properly and result is exactly as it should be – that way, any chance you could point to the resources I could look at? Thanks!

    P.S. Filip I am looking to implement json parsing through extendscript solely, but thanks for a suggestion expression-wise!

  • Andrei Popa

    September 11, 2021 at 2:22 pm

    If you read A from file, you should B = eval(A) to get the object. Then you should use B.toSource() only if you modified something and want that to be saved back inside the file.

  • Tomas Bumbulevičius

    September 12, 2021 at 7:43 am

    Andrei – thanks a lot for your response. Thats exactly what I tried in the first place – but it appeared to be that eval([string variable that is read from a file line by line]) doesn’t return anything. That is why I tried toSource back and forth to just see whats happening there. Any ideas why eval might not work?

    I found out that adding parentheses to eval actually works:

    eval(‘(‘ + a + ‘)’)

    On the other hand, I found a different way to turn string into object, which works.

    var b = (new Function( “return ” + a ) )() ;

    But be aware if anyone uses, that its also injecting the code / not safe method, as well as eval.

    Last but not least, is there a better way to read files into string, than readln in a while eof loop? As I see that with 1MB+ files its slow as well.

    Cheers!

Viewing 1 - 10 of 11 posts

Log in to reply.

We use anonymous cookies to give you the best experience we can.
Our Privacy policy | GDPR Policy