Skip to content
AI NEWS JOURNAL
AI, IN PERSPECTIVE

Search AI news

Search headlines, summaries and articles to connect the dots.

Newest first

GitHub Copilot in JetBrains, Ignite releases

This update brings remote access, on-device voice dictation, and browser use controls to GitHub Copilot for JetBrains. Copilot users in JetBrains IDEs also get context usage shortcuts, historical diffs, simpler… The post GitHub Copilot in JetBrains, Ignite releases appeared first on The GitHub Blog.

Vercel Blob now supports TTLs for automatic deletion

Vercel Blob now allows automatic deletion of blobs using a time-to-live (TTL) feature, where blobs are deleted automatically between 1 to 730 days after upload. TTL can be set during write operations like put(), copy(), putImage(), multipart uploads, and presigned URLs. The expiresAt timestamp is included in all write results and head() responses. TTL can be modified or removed later using setTtl() without re-uploading. CLI commands support --ttl-days and --clear-ttl flags. Setting a TTL incurs one Advanced Operation per 60 days, with costs scaling up to 13 for the maximum 730-day TTL. Deletion and clearing TTL are free. Usage can be tracked via the Observability tab.

Agents are users, so your software needs evals

Agents interact with software through a funnel similar to humans: Discovery, Selection, Access, and Success. To measure whether agents complete user tasks correctly, end-to-end (E2E) evaluations are essential. These tests run multiple trials, score outcomes, and help identify failures across steps like searching cafes, checking budgets, and placing orders. Using frameworks like Harbor, teams can define tasks with specific instructions, environments, verifiers, and known-good solutions. Metrics such as success rate, Pass@k, and Pass^k reveal how reliably agents perform, while trace analysis uncovers root causes of failure. The process includes setting a goal based on observability, customer feedback, or roadmap needs; defining measurable metrics; designing task sets covering common and edge cases; running evaluations at scale using tools like Vercel Sandbox and AI Gateway; and analyzing traces to iterate improvements. Automated trace analysis enables scaling, and repeated evaluation supports continuous improvement, regression detection, and product optimization.

WTF are evals? How to test agents and the software they use

Agents interact with software through a funnel similar to humans: Discovery, Selection, Access, and Success. To measure whether agents complete user tasks correctly, end-to-end (E2E) evaluations are essential. These evaluations assess the full sequence of steps an agent takes, ensuring outcomes meet user requirements. Using frameworks like Harbor, developers can design tasks with defined environments, verifiers, and known-good solutions, then run trials across variants to measure success rate, pass@k, pass^k, efficiency, time, and cost. Analyzing traces reveals root causes of failures, enabling targeted improvements. Iterative evaluation helps catch regressions, test changes, and guide product development, turning evaluations into ongoing infrastructure for building better agent-driven products.