How to improve pyspark script performance? It has huge data processing in it.

Question

I have written one pyspark script. In which I'm extracting few required fields from nested json structure. And creating dataframe on top of it. But it is taking to kuch time to process even less amount of records.

Answer 1

without having seen your code we can not help you. you can put your code in playground and link it here. Thanks!

Answer 2

You may want to look into threading if you have too many processes going on.