Refactor Your Python Stock Data Fetching Script for Better Maintainability and Readability

Refactoring the Code

The original code can be refactored to improve its structure and maintainability. Here’s a revised version:

import requests
import json

# Define constants
URL = "https://www.stockrover.com/research/all/313/s_42"
API_headers = {
    'authority': 'www.stockrover.com',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"',
    'x-csrf-token': 'fAeVScD26lby5MQf5YFI5p3snudo3E+rw0TL0h1W3j/vcjsIMvgxAF5Z9DkMjjCU4trT/b4EV0VCCPvmms5VIw==',
    'sec-ch-ua-mobile': '?0',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36',
    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'accept': 'application/json',
    'x-requested-with': 'XMLHttpRequest',
    'sec-ch-ua-platform': '"Windows"',
    'origin': 'https://www.stockrover.com',
    'sec-fetch-site': 'same-origin',
    'sec-fetch-mode': 'cors',
    'sec-fetch-dest': 'empty',
    'referer': 'https://www.stockrover.com/research/all/313/s_42/ANDE',
    'accept-language': 'en-US,en;q=0.9',
    'cookie': 'remember_me_pref=0; user_name=test11964; plan=3; premiumBraintreeKey=MIIBCgKCAQEAzM4LJfrNnBOgRFB1dDJkmqTFCWT2Y+ksOydD8xDH4R033WUzxbffMZb+3dqEyQvOVjLcwFIHByDc4Xwej7enas2E/VRyh7Cvyadn7M5zQeRyLcI9Ys5KCozMwxJPc0x76FlXPwiAo1Qlz3RcLb9wGHBag2R51FuTie+BhVDCgzWajqDCREzRhi/qlt3D/xXNo/iwJlpOUr/x1QnkkILxgKlq1dD7KJ767O5ojYKXsO/V2Bfu7sSD3djsOxQJ1/RbaDm2E96EDkWhhOeOpPndQ6IuSl4NmnJg/cq6f8csW8M3Ys/MZPFkdxPC4/fRM1XC9o76PjpVNBIO/byJEELKZedwIDAQAB; lr=1644876886; _Ruby2_session=OEs3djBGVmtrSXhSWjFibnp2ck9vem8vd1VmQ00wUkZCVWlaWmYzOFBQQUJyemc0MFNjMlVmTGRUeFpNSTFIRGliVitnb2M1L2JNcCt5SnQxN2xtZDV5M20waEd0elh3aUU3R0k3YnJiVy9BcUhObGpBUU54c2tQRE9RZHBZWk8wa0NBQXJub2tHU3pHZmUvM3dRNGZvVStsT0ZEbUNEa3ZyVDkxdDA5S1B5d3VZWlZNUERxL01VNlYzdHhBci9uSnduOGxodk8rSDJLaUJIWTVyM25GL3o4RHYva3pGeDdIY1NJODV4WkV4MnRiR2RoNHZDUktPWSsxaElPdXNEU0tSaCtKL1o0K2FsSE1rcVI2TkgxOFo0MktSL1JRWjgzRllBZGFhMjg4bitpTUJzQjR6Nk5OZzhzUFMzVGRpVFNZOUxrTnhWNTB2K0dYNXdDcFFXZnpZdlZleFR2cWY5cmJqaDNiY0JhWVJJT0s1TDEvOHU0UTV6NW5uTjcwZjFzdHpxeFg0cVQ3NTRob2xTMlA2ZDhCQT09LS1LOEljN3laVTlBVUpsVVo3c093Y2NnPT0=--b963330daa985315420ea5893f1cfa3e3a54c9d5; lr=1644876939'
}

def fetch_data(page, limit):
    payload = {
        'page': page,
        'start': page * limit,
        'limit': limit
    }
    response = requests.request("POST", URL, headers=API_headers, data=payload)
    return response.json()['stock_infos']

def process_data(data):
    stocklist = [data]
    flattened = [item for sublist in stocklist for item in sublist]
    return flattened

def main():
    stock_info_list = fetch_data(0, 250) # Adjust limit to your desired number of records
    processed_stock_info = process_data(stock_info_list)
    stock_data(processed_stock_info)

if __name__ == "__main__":
    main()

Key Changes:

Constants: Defined URL and API_headers as constants at the top for better readability and maintainability.
Modularization: Extracted functions to fetch data, process it, and handle the stock data processing logic into separate modules (fetch_data, process_data, main).
Data Fetching: Modified the fetch_data function to take two parameters: page number and limit. This allows for dynamic fetching of records based on the user’s preference.
Processing Data: Introduced a new function called process_data that takes the fetched data as input, flattens it into a single list, and returns this processed data.
Main Functionality: Created a main function (main) that ties everything together by fetching initial stock information, processing it, and then calling the stock_data function.

Example Use Case:

To change the number of records fetched, you can adjust the limit parameter when calling the fetch_data function in the main method. For example:

stock_info_list = fetch_data(0, 500) # Fetches 500 records

This would result in fetching and processing a total of 500 stock information items.

Last modified on 2023-10-27