Building on AWS

Building on AWS M 917 536 3378 maksim_kozyarchuk@yahoo.com

---

When I started developing the Kupala-Nich platform, I knew I wanted to leverage AWS’s serverless environment. New to serverless technology, I spent considerable time refining the platform’s architecture, learning the limitations of certain tools, and discovering AWS components along the way. Here, I’ll share the current architecture of the platform and invite feedback on whether best practices or AWS stack components could further enhance its robustness, security, and efficiency. Below is a high-level diagram of the Kupala-Nich application architecture followed by a brief description of the components. Platform Architecture Overview The frontend is a React application that maintains a WebSocket connection to the API Gateway, handling most server interactions. The backend consists of several Lambda functions, DynamoDB, and S3 storage. WS Lambda: A lightweight Lambda function that exposes various endpoints for the UI. Built in Python without dependencies beyond the standard library, on average it responds in under 50ms and requires less than 100MB of RAM. PyCaret Lambda: Deployed via Docker, this function packages PyCaret for ML analysis, taking 3-5 minutes to execute with ~500MB of RAM. Training datasets and generated analysis are stored in S3. CalcPosition Lambda: Triggered by DynamoDB Streams upon updates to the Position and Market Data tables, it calculates positions and P&L values, updates the CalcPosition table, and publishes results to WebSocket clients. Although light, it will scale as position complexity increases. EOD & YFinance Lambdas: These are event-triggered by timers. YFinance refreshes market data, and EOD snapshots positions to the EODPosition table and performs maintenance, such as rebalancing and closed position aggregation. The YFinance Lambda requires pandas and yfinance libraries, so it’s deployed as a zip package. Code Repositories & CI/CD Pipeline The platform’s complexity lies not only in its architecture but in the automation of CI/CD pipelines and management of permissions. Here’s an overview of the code structure and CI/CD practices: Frontend Repo: This contains all React code. The CI/CD pipeline is straightforward, running npm install, tests, npm build, and finally deploying the build to the Apache server. The pipeline, built on Node, completes in about two minutes, with most time in npm install and build steps. Unit test coverage is moderate, focusing on formatting logic and component stability across changes. Backend Repo: This includes all backend code, except for PyCaret-specific functions, which are in a separate repository. Each Lambda has its own Python package with shared components in a common package. Tests coverage is extensive, most integration-level scenarios are using Moto for AWS mocking. Lightweight Lambdas share a package with different entry points based on the trigger type. YFinance Lambda is packaged separately due to dependencies. The repo also includes a CloudFormation template that defines the application’s tables, Lambdas, API Gateway configurations, and security roles. CI/CD here includes a test stage for type checking and validation and a deployment stage for package and SAM deployments. PyCaret Repo: This houses code for PyCaret-based analysis and data retrieval. The Lambda can be triggered by WebSocket API Gateway or AWS Step Function events. Test coverage is minimal as the focus is on PyCaret invocations. Docker packaging takes about 7 minutes due to PyCaret’s dependencies. To expedite testing, different base images are used for test and package steps. This repo also includes two CloudFormation templates, one for the Lambda and another for the AWS Step Function definition. Next Steps: Scalability & Security As the Kupala-Nich platform evolves from a demo to a production product, scaling and security will be crucial focuses. I’ll cover these topics in detail in a future article. If you would like access to the code repo, please reach out to me by email.