Implemented trading time restrictions to limit engine activity to market hours and added logic to prune the universe of instruments known to have problematic error types (e.g., KID). Data handling was further robustified by migrating from Excel to CSV for the master price series and improving broker error suppression.

This significant refactor formalizes conId as the canonical identifier across the codebase, replacing symbol-based resolution to improve data stability. It cleanly separates candidate population from universe validation, enabling an append-only candidate list while allowing the live universe to shrink dynamically. Additionally, the enrichment schema was tightened to 8 essential fields to simplify data management.

This update introduces a formal universe refresh mechanism, allowing the model to validate candidates against broker data at startup. It also improves robustness in the data pipeline by adding atomic, crash-safe file writing and corruption quarantine for master price files, preventing data loss during interrupted runs. Finally, improved broker connection management ensures IBKR session consistency by caching and explicitly closing connections, reducing the likelihood of session reuse errors.

We have finalized the migration away from IG by removing all remaining artifacts, including deprecated adapters, stale configuration data, and broken test suites. This cleanup involved deleting over 10,000 lines of legacy code and data files, effectively hardening the codebase around the current IBKR implementation.

This update significantly refactors the codebase by removing standalone diagnostic scripts (datasource_scoper.py, discover_universe.py) in favor of a new, unified diagnostic_tools/refresh_universe.py. This new tool streamlines instrument discovery and validation into a single, automated, and memory-driven API pipeline. As part of this cleanup, several legacy input files were removed, effectively consolidating the infrastructure for managing the trading universe.

This update resolves a JSONDecodeError during incremental fetches by correcting the IG API date format normalization from slash-separated to dash-separated. Additionally, I have introduced new skills to the documentation, including a Bayesian+Socratic bug-diagnosis agent (Fixer.md) and a distillation of bugfixing principles and conversation indexing, all designed to streamline future troubleshooting workflows.

We identified that the discover_universe process was unnecessarily retrying deterministic epic.unavailable errors from the IG API, wasting roughly 15 minutes per run and hitting API limits. I introduced a fast-fail mechanism to ignore these errors immediately and ensured they are correctly classified as EPIC_NOT_RECOGNIZED instead of generic API_ERROR. This change saves significant time and prevents cascading failures during the discovery phase.

Updated the universe discovery tools to streamline the ingestion and two-tier validation of trading instruments. The pipeline now automates candidate registration and metadata validation, ensuring only viable instruments are passed to the trading engine.

We significantly improved our data visibility by expanding the EPIC_TO_YH_TICKER mapping from 34 to 97 entries, covering nearly the entire T1-PASS universe. This update also adds the yh_ticker as column B in the candidates_report, ensuring better source transparency. Additionally, we hardened the data pipeline by updating the _safe() function to properly handle inf/-inf values.

Updated the instrument universe configuration to include a diverse set of new forex, commodity, and cryptocurrency pairings. To maintain stability with the broker API, the rate limit delay was increased from 0.5 to 2 seconds, and automated discovery was enabled by default in the main configuration.

To prevent data gaps when the IG broker adapter fails to return usable information, we have introduced a YHFinanceFetcher as a transparent secondary data source. The pipeline now automatically retries failed data fetches via Yahoo Finance, tracks source usage, and generates a new candidates_report.csv to provide better visibility into data availability across the universe.

Added a new dataset, data/input/ig_epic_candidates.txt, containing 322 researched IG epic codes across various asset classes including indices, equities, forex, commodities, and crypto. This replaces the previous small set of hand-picked candidates, providing a robust seed list for the discover_universe.py tool. These candidates will be validated via the IG Demo API to ensure live data availability before being added to the final trading universe.

DataPipeline was fetching and persisting unused bid_close and mid_open price series in the universe_series.xlsx file. This refactor removes those fields entirely, simplifying the pipeline and reducing storage footprint. Note that existing universe_series.xlsx files will automatically have these stale sheets dropped upon the next successful data ingestion run.

This update adds a dedicated 'Inputs' section to the README, clarifying how instrument universes and price series are managed, validated, and merged. It also corrects several high-priority inaccuracies regarding merge precedence, discover_universe.py behavior, and configuration parameters.

Reduced the data lookback period from previous settings to 5 bars in main.py to optimize processing. Also included updates to the universe series configuration and updated the dashboard screenshot in media assets.

Implemented a new GitHub Pages delivery mode that automatically pushes the dashboard_data.json file to a repository using the GitHub Contents API. This allows for easier remote dashboard hosting by treating GitHub as an automated file-sync host for the monitoring pipeline.

Previously, the DASHBOARD_DATA_URL configuration was silently ignored when using Netlify delivery, forcing the system to always use a local relative filename. I've updated the performance monitoring logic to correctly respect the configured data URL for both FTP and Netlify modes. This ensures custom data URLs are properly handled during automated deployments.

Updated the universe_series.xlsx file containing core domain data. This change ensures the application is using the latest dataset for universe-related calculations and queries.

We've introduced a new deployment mode that allows you to host the performance dashboard on Netlify, providing an excellent alternative for users who do not have access to traditional cPanel/FTP hosting. This implementation leverages Netlify's Files Deploy API using only standard libraries, ensuring an efficient, incremental upload process that only pushes changed files. Setup is straightforward and documented in the updated hosting guide.

Updated the performance monitoring dashboard to use a relative URL for its data source (dashboard_data.json) when operating in localhost or file_only delivery modes. This modification ensures that the front-end correctly hydrates when the dashboard HTML file is opened directly from the filesystem or a local server, preventing issues where hardcoded absolute paths might fail to resolve.