Pandas - How To Load Data From Nested Dictionary Into Dataframe?

I'm tryping to create a dataframe with closingprices for stocks and have found a free api that returns JSON-data in the form of nested dicts with the data, looking like this: {'nam

Solution 1:

If you just want to load it in dataframe:

# h = your dictionarydf = pd.DataFrame.from_dict(data=h['history'],orient='index')

cols = ['close']

df = df[cols]

# Just as an aside Quandl has been very good for free financial data to me. #It has a paid side with premium data but I havent used it.

Solution 2:

You don't need to know which key is present to access it. You can just iterate over all the keys in the dictionary.

d = <your dict>
retval = {}

for k,v in d['history'].items():
    retval[k] = v['close']

print(retval)

Solution 3:

If you know your keys, and they don't change, I would use Droids answer. If the keys may change here is a different solution.

d = {'name': 'AAPL',
 'history': 
           {'2019-01-04': 
                  {'open': '144.53',
                   'close': '148.26',
                   'high': '148.55',
                   'low': '143.80',
                   'volume': '58607070'},
           '2019-01-03': 
                   {'open': '143.98',
                    'close': '142.19',
                    'high': '145.72',
                    'low': '142.00',
                    'volume': '91312195'},
          '2019-01-02': 
                  {'open': '154.89',
                   'close': '157.92',
                   'high': '158.85',
                   'low': '154.23',
                   'volume': '37039737'
}}}
defprint_nested_dict(nested_dict, name, prior_keys=[]):
    for key, value in nested_dict.items():
        # current_key_path is a list of each key we used to get here
        current_key_path = prior_keys + [key]
        # Convert that key path to a string
        key_path_str = ''.join('[\'{}\']'.format(key) for key in current_key_path)

        # If the value is a dict then recurseifisinstance(value, dict):
            print_nested_dict(value, name, current_key_path)
        else:
            # Else lets print the key and value for this value# along with where it was foundprint(key, value, '{}{}'.format(name, key_path_str))
print_nested_dict(d, "d")

Output:

name AAPL d['name']
open 144.53 d['history']['2019-01-04']['open']
close 148.26 d['history']['2019-01-04']['close']
high 148.55 d['history']['2019-01-04']['high']
low 143.80 d['history']['2019-01-04']['low']
volume 58607070 d['history']['2019-01-04']['volume']
open 143.98 d['history']['2019-01-03']['open']
close 142.19 d['history']['2019-01-03']['close']
high 145.72 d['history']['2019-01-03']['high']
low 142.00 d['history']['2019-01-03']['low']
volume 91312195 d['history']['2019-01-03']['volume']
open 154.89 d['history']['2019-01-02']['open']
close 157.92 d['history']['2019-01-02']['close']
high 158.85 d['history']['2019-01-02']['high']
low 154.23 d['history']['2019-01-02']['low']
volume 37039737 d['history']['2019-01-02']['volume']

That being said, there may be a more efficient way then this using built in dataframe methods.

Solution 4:

You can use a regular expression:

import re

if re.match(r"^(\d+-\d+-\d+)$", key):
    # do something with it's values.

You will need to loop over the dictionary yourself, however.