docs: update readme and coding guidelines

.gitignore

@@ -0,0 +1,49 @@
+# General
+*.log
+.DS_Store
+*.pem
+.env*
+!.env.example
+*.iml
+.idea/
+.vscode/
+
+# Python / Backend
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+*.sqlite3
+*.db
+.venv/
+/backend/.venv/ # Explicitly ignore nested venv if backend is subdir
+
+# Node / Frontend
+node_modules/
+.next/
+out/
+build/
+.pnp.*
+.yarn/*
+!.yarn/patches
+!.yarn/plugins
+!.yarn/releases
+!.yarn/versions
+coverage/
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+*.tsbuildinfo
+next-env.d.ts
+.vercel/
+
+# Uploads (ensure this matches your config.py UPLOAD_DIR)
+/backend/uploads/
+# If uploads are outside backend:
+# /uploads/

CODING_GUIDELINES.md

@@ -0,0 +1,82 @@
+# Coding Guidelines & Standards
+
+These guidelines aim to ensure code consistency, readability, and maintainability across the Borbann project for both frontend (TypeScript/React) and backend (Python/FastAPI).
+
+## 1. Formatting
+
+- **General:** Code should be consistently formatted. Use automated tools to enforce this.
+- **Frontend (TypeScript/React):**
+  - Use **Prettier** for code formatting. Configure it (`.prettierrc`) and integrate it with your IDE and pre-commit hooks.
+  - Use **ESLint** for linting. The project already has `eslint.config.mjs` based on `next/core-web-vitals`. Ensure rules are agreed upon and enforced.
+- **Backend (Python):**
+  - Use **Black** for code formatting.
+  - Use **Ruff** (or Flake8 + isort) for linting and import sorting. Configure it in `pyproject.toml`.
+- **Line Length:** Aim for a maximum line length of ~100-120 characters for better readability, but prioritize clarity over strict adherence. Formatting tools usually handle this.
+- **Indentation:** Use 4 spaces for Python, 2 spaces for TypeScript/JSX/CSS/JSON (or configure Prettier as desired). **Do not use tabs.**
+
+## 2. Naming Conventions
+
+- **General:** Use descriptive and meaningful names. Avoid abbreviations unless they are widely understood (e.g., `id`, `url`, `http`).
+- **Frontend (TypeScript/React):**
+  - **Components:** `PascalCase` (e.g., `UserProfileCard.tsx`).
+  - **Variables/Functions:** `camelCase` (e.g., `fetchUserData`, `userName`).
+  - **Constants:** `UPPER_SNAKE_CASE` (e.g., `const MAX_USERS = 100;`).
+  - **Interfaces/Types:** `PascalCase` (e.g., `interface UserProfileProps`).
+  - **Hooks:** Prefix with `use` (`camelCase`, e.g., `useAuthStatus`).
+  - **Files:** Match the main export (e.g., `UserProfileCard.tsx` for `UserProfileCard` component). Use `kebab-case` for utility/config files (e.g., `api-client.ts`).
+- **Backend (Python):**
+  - **Variables/Functions/Methods:** `snake_case` (e.g., `fetch_user_data`, `user_name`).
+  - **Classes:** `PascalCase` (e.g., `class DatabaseManager:`).
+  - **Constants:** `UPPER_SNAKE_CASE` (e.g., `MAX_RETRIES = 3`).
+  - **Modules/Packages:** `snake_case` (e.g., `data_processing_service.py`).
+
+## 3. Basic Principles
+
+- **KISS (Keep It Simple, Stupid):** Prefer simple solutions over complex ones.
+- **DRY (Don't Repeat Yourself):** Avoid code duplication. Use functions, components, classes, or modules to encapsulate reusable logic.
+- **YAGNI (You Aren't Gonna Need It):** Avoid implementing functionality until it's necessary.
+- **Readability:** Write code that is easy for others (and your future self) to understand. Use clear variable names, add comments where necessary, and structure code logically.
+- **Modularity:** Break down code into smaller, reusable, and testable units (functions, components, classes, modules).
+- **Comments:**
+  - Explain _why_ something is done, not _what_ it does (the code should explain the 'what').
+  - Comment complex logic, workarounds, or important assumptions.
+  - Keep comments up-to-date with the code.
+  - Use TODO/FIXME comments judiciously for temporary notes.
+  - **Python:** Use docstrings for modules, classes, and functions (`"""Docstring goes here."""`).
+  - **TypeScript:** Use TSDoc comments (`/** Doc comment here */`) for functions, classes, types, and complex logic.
+
+## 4. Error Handling
+
+- **General:** Avoid silent failures. Handle potential errors gracefully.
+- **Backend (Python):**
+  - Use specific exception types (e.g., `ValueError`, `TypeError`, custom exceptions) rather than generic `Exception`.
+  - Log errors effectively, including relevant context (e.g., user ID, request ID).
+  - Use `try...except...finally` blocks appropriately.
+  - Provide meaningful error responses in APIs (e.g., using FastAPI's `HTTPException`).
+- **Frontend (TypeScript/React):**
+  - Use `try...catch` for asynchronous operations (API calls).
+  - Implement Error Boundaries for React components to catch rendering errors.
+  - Provide clear feedback to the user when errors occur (e.g., toast notifications, inline messages).
+  - Validate user input thoroughly.
+
+## 5. Testing
+
+- **General:** Write tests for your code.
+  - **Unit Tests:** Test individual functions, components, or classes in isolation.
+  - **Integration Tests:** Test the interaction between different parts of the system (e.g., API endpoint calling a service that uses the DB).
+  - **End-to-End (E2E) Tests:** Test the application flow from the user's perspective.
+- **Backend:** Use `pytest`. Aim for good test coverage, especially for business logic and CRUD operations. Use fixtures for setup/teardown. Mock external dependencies where appropriate.
+- **Frontend:** Use Jest/Vitest with React Testing Library for unit/integration tests. Consider Cypress or Playwright for E2E tests.
+
+## 6. Imports
+
+- **Frontend:** Use absolute imports (`@/components/ui/button`) over relative imports (`../../components/ui/button`) where possible (already configured via `tsconfig.json`). Keep imports organized (e.g., React imports first, then external libs, then internal modules/components). ESLint can help enforce this.
+- **Backend:** Organize imports according to PEP 8 (stdlib, third-party, local application). Use absolute imports within the application (e.g., `from app.services import ...`). Ruff/isort handles this automatically.
+
+## 7. Version Control
+
+- Use Git for version control.
+- Follow the agreed commit message convention (see separate document).
+- Make small, atomic commits.
+- Use feature branches for development and Pull Requests (PRs) for merging.
+- Require code reviews for PRs.

COMMIT_CONVENTION.md

@@ -0,0 +1,96 @@
+# Commit Message Convention
+
+This project follows the **Conventional Commits** specification (v1.0.0). This leads to more readable messages that are easier to follow when looking through the project history and allows for automated changelog generation.
+
+Reference: [https://www.conventionalcommits.org/](https://www.conventionalcommits.org/)
+
+## Format
+
+Each commit message consists of a **header**, a **body** and a **footer**.
+
+```
+<type>[optional scope]: <description>
+
+[optional body]
+
+[optional footer(s)]
+```
+
+## Header
+
+The header is mandatory and includes a **type**, an optional **scope**, and a **description**.
+
+### `<type>`
+
+This describes the kind of change that this commit is providing. Must be one of the following:
+
+- **feat**: A new feature
+- **fix**: A bug fix
+- **refactor**: A code change that neither fixes a bug nor adds a feature
+- **perf**: A code change that improves performance
+- **style**: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
+- **test**: Adding missing tests or correcting existing tests
+- **build**: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
+- **ci**: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
+- **docs**: Documentation only changes
+- **chore**: Other changes that don't modify src or test files (e.g., updating dependencies, project config)
+- **revert**: Reverts a previous commit
+
+### `<scope>` (Optional)
+
+The scope provides contextual information to the commit type. It's contained within parentheses, e.g., `feat(api): ...` or `fix(ui): ...`. Choose a scope that represents the area of the codebase affected (e.g., `auth`, `map`, `pipeline`, `worker`, `deps`, `config`, `ci`, `ui`, `docs`).
+
+### `<description>`
+
+A short summary of the code changes.
+
+- Use the imperative, present tense: "change" not "changed" nor "changes".
+- Don't capitalize the first letter.
+- No dot (.) at the end.
+
+## Body (Optional)
+
+The body should include the motivation for the change and contrast this with previous behavior. Use the imperative, present tense.
+
+## Footer(s) (Optional)
+
+- **Breaking Changes:** Start with `BREAKING CHANGE:` followed by a description of the breaking change.
+- **Referencing Issues:** Use keywords like `Closes #123`, `Fixes #456`.
+
+## Examples
+
+```
+feat: add user authentication endpoint
+```
+
+```
+fix(api): correct pagination logic for pipelines endpoint
+
+The previous implementation did not handle the offset correctly,
+causing duplicate items on subsequent pages.
+```
+
+```
+refactor(map): extract overlay management into separate context
+
+Improves separation of concerns and makes the map component cleaner.
+```
+
+```
+chore(deps): update FastAPI to version 0.100.0
+```
+
+```
+test(crud): add unit tests for pipeline run creation
+```
+
+```
+docs: update README with deployment instructions
+```
+
+```
+feat(ui)!: replace Button component with new design system button
+
+BREAKING CHANGE: The `variant` prop for Button components has changed.
+Refer to the updated design system documentation. Closes #78
+```

README.md

@@ -0,0 +1,202 @@
+# Borbann Project
+
+A data integration and analysis platform focused on real estate.
+
+## Project Structure
+
+- `/backend`: Contains the Python FastAPI backend application, background workers, and database models.
+- `/frontend`: Contains the Next.js (React/TypeScript) frontend application.
+
+## Prerequisites
+
+Ensure you have the following installed on your system:
+
+- **Node.js:** (Specify version, e.g., v20.x or later)
+- **pnpm:** (Specify version or `npm install -g pnpm`)
+- **Python:** (Specify version, e.g., v3.11 or v3.12, matching `.python-version` if present)
+- **Docker:** Latest version
+- **Docker Compose:** Latest version (often included with Docker Desktop)
+- **Database Client:** (Optional, for direct DB access, e.g., `psql` for PostgreSQL)
+
+## Environment Setup
+
+Both frontend and backend might require environment variables.
+
+1.  **Backend:**
+    - Navigate to the `backend` directory: `cd backend`
+    - Create a `.env` file by copying `.env.example`: `cp .env.example .env`
+    - **Crucially, fill in the required values in `.env`**, especially secrets like database passwords and API keys. **Never commit your `.env` file to Git.**
+2.  **Frontend:**
+    - Navigate to the `frontend` directory: `cd frontend`
+    - Create a `.env.local` file if needed (e.g., for `NEXT_PUBLIC_API_URL`). Copy from `.env.example` if one exists. **Do not commit `.env.local` files containing secrets.**
+
+_(See `.env.example` files in respective directories for required variables)_
+
+## Setup Instructions
+
+### 1. Backend Setup
+
+```bash
+# Navigate to the backend directory
+cd backend
+
+# Create and activate a virtual environment (recommended)
+python -m venv .venv
+# Windows:
+# .\ .venv\Scripts\activate
+# macOS/Linux:
+source .venv/bin/activate
+
+# Install dependencies (uses pyproject.toml)
+pip install -e .
+
+# Or if requirements.txt is preferred:
+# pip install -r requirements.txt
+
+# Set up the database (Run Docker Compose first if DB is containerized)
+# Example assuming Alembic migrations (add if you use Alembic)
+# alembic upgrade head
+```
+
+### 2. Frontend Setup
+
+```bash
+# Navigate to the frontend directory
+cd frontend
+
+# Install dependencies using pnpm
+pnpm install
+```
+
+## Running the Application
+
+### Development Mode
+
+You typically need multiple terminals for development.
+
+1.  **Start Docker Services (Database, Redis, etc.):**
+
+    ```bash
+    # From the 'backend' directory
+    docker-compose up -d db redis # Add other services as needed
+    ```
+
+2.  **Start Backend API:**
+
+    ```bash
+    # From the 'backend' directory (with virtual env activated)
+    uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
+    ```
+
+    _(The API will be accessible at http://localhost:8000)_
+
+3.  **Start Celery Worker (if needed for background tasks):**
+
+    ```bash
+    # From the 'backend' directory (with virtual env activated)
+    celery -A app.workers.celery_app worker --loglevel=info
+    ```
+
+4.  **Start Celery Beat (if using scheduled tasks):**
+
+    ```bash
+    # From the 'backend' directory (with virtual env activated)
+    celery -A app.workers.celery_app beat --loglevel=info --scheduler django_celery_beat.schedulers:DatabaseScheduler
+    # Or if not using DB scheduler:
+    # celery -A app.workers.celery_app beat --loglevel=info
+    ```
+
+5.  **Start Frontend:**
+    ```bash
+    # From the 'frontend' directory
+    pnpm dev
+    ```
+    _(The frontend will be accessible at http://localhost:3000)_
+
+### Production Mode (Using Docker)
+
+```bash
+# Navigate to the 'backend' directory (or project root if compose file is there)
+cd backend
+
+# Build the Docker images
+docker-compose build
+
+# Start all services in detached mode
+docker-compose up -d
+
+# To stop the services
+docker-compose down
+```
+
+## Building for Production
+
+### Backend
+
+Python code doesn't typically require a separate build step unless you are creating a distributable package. Docker handles the "build" in the context of creating an image.
+
+### Frontend
+
+```bash
+# Navigate to the frontend directory
+cd frontend
+
+# Create a production build
+pnpm build
+```
+
+This will create an optimized build in the `.next` directory. Use `pnpm start` to run this build.
+
+## Testing
+
+### Backend
+
+```bash
+# Navigate to the backend directory (with virtual env activated)
+pytest
+```
+
+_(Ensure test database is configured and any necessary test setup is performed)_
+
+### Frontend
+
+```bash
+# Navigate to the frontend directory
+pnpm test
+```
+
+_(Add specific test commands if using Jest, Cypress, etc.)_
+
+## Linting and Formatting
+
+### Backend
+
+```bash
+# Navigate to the backend directory (with virtual env activated)
+# Check formatting
+black --check .
+# Format code
+black .
+
+# Check linting/imports
+ruff check .
+# Fix linting/imports where possible
+ruff check . --fix
+```
+
+### Frontend
+
+```bash
+# Navigate to the frontend directory
+# Check linting
+pnpm lint
+
+# Check/Apply formatting (assuming Prettier is setup)
+pnpm prettier --check .
+pnpm prettier --write .
+```
+
+## Deployment
+
+- **Backend:** Can be deployed using Docker containers on services like AWS ECS, Google Cloud Run, Kubernetes, etc. Ensure environment variables are securely managed.
+- **Frontend:** The Next.js application can be easily deployed to platforms like Vercel (recommended), Netlify, AWS Amplify, or self-hosted using Node.js or Docker.

backend/.env.example

@@ -0,0 +1,17 @@
+# Backend Configuration
+# Rename this file to .env and fill in the values.
+# DO NOT COMMIT THE ACTUAL .env FILE.
+
+# Database
+DATABASE_URL=postgresql+asyncpg://user:password@db:5432/data_pipeline_db
+
+# Celery / Redis
+CELERY_BROKER_URL=redis://redis:6379/0
+CELERY_RESULT_BACKEND=redis://redis:6379/0
+
+# API Keys (Replace with actual keys or use secret management)
+OPENAI_API_KEY="your_openai_key_here_if_needed"
+NEWS_API_KEY="your_bing_news_or_other_api_key_if_needed"
+
+# Other settings if needed
+# SECRET_KEY=...