{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# šŸŽ® Glitch Media ā€” Multi-Agentic AI Pipeline\n", "\n", "> Otomatisasi media berita game: Scraping ā†’ Screening ā†’ Caption ā†’ Image ā†’ HITL ā†’ Upload\n", "\n", "Jalankan setiap sel secara berurutan. Pipeline akan berhenti di **Cell 6** untuk menunggu keputusan ACC/REJECT." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 1 ā€” Import & Load Environment\n", "\n", "Jalankan sel ini terlebih dahulu untuk mengimport semua dependencies dan load environment variables dari `.env`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "āœ… Import & environment loaded successfully.\n", "šŸ“ Project root: d:\\Amikom\\semester 6\\proyek data mining\\glitch-media\n", "============================================================\n", "MODEL CONFIGURATION\n", "============================================================\n", "\n", "LLM MODELS (Per Agent):\n", " Scraping: ag/gemini-flash-low\n", " Screening: ag/claude-sonnet-4-6\n", " Caption (X): ag/claude-sonnet-4-6\n", " Caption (IG): ag/claude-sonnet-4-6\n", " Upload (X): ag/gemini-flash-low\n", " Upload (IG): ag/gemini-flash-low\n", "\n", "IMAGE GENERATION:\n", " Model: ag/gemini-3-flash-agent\n", "\n", "DEFAULT LLM:\n", " Model: ag/gemini-pro-agent\n", " Max Tokens: 10000\n", " Temperature: 0.4\n", "\n", "šŸ”Œ 9Router:\n", " URL: http://localhost:20128/v1\n", "\n", "============================================================\n" ] } ], "source": [ "import sys\n", "import logging\n", "from pathlib import Path\n", "\n", "# Pastikan project root ada di sys.path\n", "PROJECT_ROOT = Path.cwd()\n", "if str(PROJECT_ROOT) not in sys.path:\n", " sys.path.insert(0, str(PROJECT_ROOT))\n", "\n", "# Import config (otomatis load .env)\n", "from config import settings\n", "\n", "# Setup logging\n", "settings.LOGS_DIR.mkdir(parents=True, exist_ok=True)\n", "logging.basicConfig(\n", " level=logging.INFO,\n", " format='%(asctime)s [%(name)s] %(levelname)s: %(message)s',\n", " handlers=[\n", " logging.FileHandler(settings.LOG_FILE, encoding='utf-8'),\n", " logging.StreamHandler(),\n", " ],\n", ")\n", "logger = logging.getLogger('main')\n", "\n", "# Import pipeline modules\n", "from pipeline.state import PipelineState\n", "from pipeline import orchestrator\n", "from validation import human_validation\n", "\n", "print('\\u2705 Import & environment loaded successfully.')\n", "print(f'\\ud83d\\udcc1 Project root: {PROJECT_ROOT}\\n')\n", "\n", "print(\"=\" * 60)\n", "print(\"MODEL CONFIGURATION\")\n", "print(\"=\" * 60)\n", "\n", "print(f\"\\n\\ud83d\\udd0c LLM Provider: {settings.LLM_PROVIDER.upper()}\")\n", "if settings.LLM_PROVIDER == 'cloud':\n", " print(f\" DashScope URL: {settings.DASHSCOPE_BASE_URL}\")\n", " print(f\" API Key: {'***' + settings.DASHSCOPE_API_KEY[-4:] if settings.DASHSCOPE_API_KEY else '(not set)'}\")\n", "else:\n", " print(f\" 9Router URL: {settings.NINEROUTER_BASE_URL}\")\n", "\n", "print(\"\\nLLM MODELS (Per Agent):\")\n", "agent_list = [\n", " (\"Caption X\", \"caption_x\", settings.AGENT_CAPTION_X_MODEL),\n", " (\"Caption IG\", \"caption_ig\", settings.AGENT_CAPTION_IG_MODEL),\n", " (\"Screening\", \"screening\", settings.AGENT_SCREENING_MODEL),\n", " (\"Scraping\", \"scraping\", settings.AGENT_SCRAPING_MODEL),\n", " (\"Image Gen\", \"image_gen\", settings.AGENT_IMAGE_GEN_MODEL),\n", "]\n", "for label, agent_name, primary_model in agent_list:\n", " chain = settings.get_fallback_chain(agent_name)\n", " full_chain = [primary_model] + [m for m in chain if m != primary_model]\n", " print(f\" {label:12s}: {' \\u2192 '.join(full_chain)}\")\n", "\n", "print(f\"\\nIMAGE GENERATION MODEL CHAIN:\")\n", "print(f\" {' \\u2192 '.join(settings.IMAGE_GEN_MODEL_CHAIN)}\")\n", "\n", "print(\"\\nDEFAULT LLM PARAMS:\")\n", "print(f\" Max Tokens: {settings.LLM_MAX_TOKENS}\")\n", "print(f\" Temperature: {settings.LLM_TEMPERATURE}\")\n", "\n", "print(\"\\n\" + \"=\" * 60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 2 ā€” Inisialisasi Pipeline State\n", "\n", "Buat `PipelineState` dengan `run_id` unik. Atau load dari checkpoint jika ingin resume.\n", "\n", "**āš ļø Pastikan Cell 1 sudah dijalankan terlebih dahulu!**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from datetime import datetime, timezone\n", "\n", "# ā”€ā”€ Buat run baru ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "run_id = datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')\n", "state = PipelineState(run_id=run_id)\n", "\n", "# ā”€ā”€ ATAU: Resume dari checkpoint ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "# Uncomment baris di bawah untuk resume dari checkpoint:\n", "# state = human_validation.load_checkpoint('20240101_120000') # ganti dengan run_id\n", "\n", "print(f'šŸ†” Run ID: {state.run_id}')\n", "print(f'šŸ“Š State initialized: raw_data={state.raw_data is not None}, '\n", " f'filtered={state.filtered_data is not None}, '\n", " f'caption_x={state.caption_x is not None}, '\n", " f'caption_ig={state.caption_ig is not None}')\n", "\n", "# Sumber berita yang akan di-scrape\n", "sources = settings.DEFAULT_SOURCES\n", "print(f'šŸ“° Sources: {sources}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 3 ā€” Stage 1-2: Scraping + Screening\n", "\n", "Sequential: Scraping ā†’ Screening (dedup + quality filter)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "state = orchestrator.run_stage_1_2(state, sources)\n", "\n", "print(f'\\nšŸ“„ Stage 1 (Scraping): {len(state.raw_data.items)} raw items')\n", "print(f'šŸ” Stage 2 (Screening): {len(state.filtered_data.items)} filtered items')\n", "\n", "# Preview filtered items\n", "print('\\n--- Filtered Items ---')\n", "for i, item in enumerate(state.filtered_data.items, 1):\n", " print(f'{i}. [{item.source}] {item.title} (score: {item.quality_score:.2f})')\n", " print(f' {item.summary}')\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 4 ā€” Stage 3: Caption X & IG (Parallel Fork)\n", "\n", "Caption X dan Caption IG berjalan paralel ā€” tidak saling bergantung." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "state = orchestrator.run_stage_3_fork(state)\n", "\n", "if state.caption_x:\n", " print(f'šŸ¦ Caption X untuk: {state.caption_x.news_ref_title}')\n", " print(f' {state.caption_x.caption_long[:200]}...')\n", " print()\n", "\n", "if state.caption_ig:\n", " print(f'šŸ“ø Caption IG untuk: {state.caption_ig.news_ref_title}')\n", " print(f' {state.caption_ig.caption_short[:200]}...')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 5 ā€” Stage 4: Image Generation\n", "\n", "Sequential ā€” tergantung pada caption IG (stage 3b)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "state = orchestrator.run_stage_4(state)\n", "\n", "if state.image_result:\n", " print(f'šŸ–¼ļø Image generated: {state.image_result.image_path}')\n", " print(f' Prompt: {state.image_result.image_prompt[:150]}...')\n", " \n", " # Tampilkan gambar di notebook\n", " from IPython.display import display, Image as IPImage\n", " try:\n", " display(IPImage(filename=state.image_result.image_path, width=400))\n", " except Exception as e:\n", " print(f' āš ļø Tidak bisa menampilkan gambar: {e}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 6 ā€” Stage 5: Human-in-the-Loop Validation\n", "\n", "āš ļø **Pipeline BERHENTI di sini** menunggu keputusan ACC/REJECT.\n", "\n", "State di-checkpoint sebelum validasi ā€” bisa di-resume jika notebook terputus." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Simpan checkpoint sebelum gate HITL\n", "human_validation.save_checkpoint(state)\n", "result = await human_validation.request_validation(state)\n", "print()\n", "\n", "# Tampilkan preview dan minta keputusan\n", "# āš ļø Pipeline akan BERHENTI di sini sampai tombol ACC/REJECT ditekan\n", "result = human_validation.request_validation(state)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 7 ā€” Stage 6: Upload (jika ACC)\n", "\n", "Jika ACC ā†’ Upload X dan IG secara paralel.\n", "Jika REJECT ā†’ Log rejection, pipeline berhenti." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if result.status == 'ACC':\n", " print('āœ… Konten di-APPROVE. Memulai upload...')\n", " print()\n", " state = orchestrator.run_stage_6_upload(state)\n", " print()\n", " print('šŸŽ‰ Upload selesai!')\n", "else:\n", " print('āŒ Konten di-REJECT.')\n", " if result.reviewer_note:\n", " print(f'šŸ“ Catatan reviewer: {result.reviewer_note}')\n", " print()\n", " print('šŸ’” Opsi:')\n", " print(' 1. Jalankan ulang dari Cell 4 (re-generate caption)')\n", " print(' 2. Jalankan ulang dari Cell 3 (re-scrape dan re-screen)')\n", " print(' 3. Edit caption secara manual dan jalankan Cell 7 lagi')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Cell 8 ā€” Ringkasan Hasil" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.display import display, HTML\n", "\n", "display(HTML('

šŸ“Š Ringkasan Pipeline Run

'))\n", "\n", "print(f'šŸ†” Run ID: {state.run_id}')\n", "print(f'šŸ“„ Raw items scraped: {len(state.raw_data.items) if state.raw_data else 0}')\n", "print(f'šŸ” Filtered items: {len(state.filtered_data.items) if state.filtered_data else 0}')\n", "print()\n", "\n", "if state.caption_x:\n", " print(f'šŸ¦ Caption X: {state.caption_x.news_ref_title}')\n", "if state.caption_ig:\n", " print(f'šŸ“ø Caption IG: {state.caption_ig.news_ref_title}')\n", "if state.image_result:\n", " print(f'šŸ–¼ļø Image: {state.image_result.image_path}')\n", "print()\n", "\n", "if state.validation:\n", " print(f'āœ… Validation: {state.validation.status}')\n", "print()\n", "\n", "print('--- Run Log ---')\n", "for log_entry in state.logs:\n", " print(f' {log_entry}')" ] } ], "metadata": { "kernelspec": { "display_name": "venv (3.10.6)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" } }, "nbformat": 4, "nbformat_minor": 4 }