Artificial Intelligence

Insights

Introduction

Generative AI is becoming popular among SaaS companies, finding uses in many areas from improving customer experience to automating processes and enhancing product discovery. Postman launched their Gen AI app, POSTBOT, which is designed to debug and interpret APIs, expedite test writing, and analyze large datasets.

Postman, a major player in the global Indian SaaS ecosystem, is known for being at the forefront of technology. The power of generative AI lies in its ability to understand natural language prompts and generate scripts for API requests with incredible speed and accuracy. It helps software engineers in Streamlining test script execution, optimizes testing processes and enables quicker product releases.

Empowering non-technical teams to contribute meaningfully to quality assurance, PostBot has proven to be an indispensable tool. Generative AI (GenAI) technologies offer a powerful solution for QA teams to achieve more with fewer resources in the software testing arena. Although test automation already aids many companies in optimizing their testing workflows, AI technologies can significantly enhance this by speeding up test creation and bolstering the comprehensiveness of testing practices. This article delves into the role of GenAI in software testing, examining its benefits in detail and outlining strategies for developing an effective QA strategy.

What is POSTMAN?

Postman is a popular API platform that simplifies the process of building, testing, and using APIs. It's a valuable tool for Developers, Testers, Consumers who work with APIs, providing a user-friendly interface for sending and receiving requests, managing collections of requests, and automating testing workflows. Postman supports the entire API life cycle management by providing a comprehensive set of tools and features to design, test, document, monitor, collaborate, secure, and integrate your APIs.

What is POSTBOT?

Postman has introduced PostBot, an AI Assistant designed to accelerate API Testing tasks with Gen AI capabilities. Postman Postbot is a cutting-edge tool that leverages artificial intelligence to augment API testing within the Postman ecosystem. By harnessing large language models (LLMs) ,advanced machine learning, Prompt Engineering (PE) and natural language processing (NLP), Postbot maximizes its effectiveness, provides intelligent assistance and streamline the API testing process. It offers several key functionalities, including automatic test generation, debugging, documentation, and analyzing large datasets.

POSTBOT LLM @ a Glance

Postman Postbot, An LLM-Powered API Assistant. Postman Postbot is a powerful AI tool that leverages the capabilities of Large Language Models (LLMs) to enhance API testing within the Postman ecosystem. Postbot built on pre-trained model from OpenAI (like GPT-3) and then fine-tune it further on a custom dataset managed through the Hugging Face platform, Model and Dataset: Developed and maintained using the Hugging Face platform and deployed on AWS SageMaker.

Figure 1: Pretrained Model to Fine Tuning Model to Final Model

POSTBOT AI Assistant Features

Postman’s AI feature leverages machine learning algorithms to analyze API responses, predict potential issues, and suggest improvements. It offers intelligent insights based on historical data, patterns, and best practices gleaned from millions of API requests across the Postman ecosystem. Listed below POSTBOT features:

Figure 2: POSTBOT AI Features

Test Script Generation

PostBot feature generates automated tests for the API endpoints. PostBot analyzes the API definition and uses AI algorithms to generate tests based on common test scenarios and user-defined testing requirements. PostBot automatically generates test scripts that validate response status codes, response body structures, data types, and specific field-level validations. It also suggests tests for error responses and edge cases.

Postbot enables the users the flexibility to customize and add additional tests as per their specific requirements. Postman provides an intuitive user interface where users can interact with the PostBot AI and instruct it to generate specific tests or modify existing ones. With the ability to write custom tests, users can address unique test scenarios, specific business logic validations, or edge cases that are not covered by the automated tests generated by PostBot.

Figure 3: Test Script Generation

FIX Broken Tests

Struggling with a failing test? Postbot AI is here to save the day! It swiftly diagnoses and rectifies syntax and structural issues, ensuring your tests run smoothly. When any assertion fails, you can invoke Postbot to update the test script. This allows you to write simple scripts initially and utilize Postbot to adjust existing code when a new endpoint response is required. You can use the "Fix test" feature to let Postbot correct the failing test. Postman's changelog is readily available for reverting any changes.

Figure 4: FIX Test feature in Test Results

Figure 5: Post using Fix Test

Save a Field from Response

Postbot is your intelligent testing assistant! It meticulously analyzes your request, draws insights from past examples, and generates precise test code. Extracted values are neatly stored in Postman environment variables, ready for your next testing adventure.

Figure 6: Quickly set a variable from Response Data

Smart Suggestions

Postman's Postbot utilizes its advanced language model to offer intelligent and contextually appropriate suggestions within the test editor. As you begin typing your tests, Postbot proactively suggests common test behaviors. Additionally, it analyzes the current API response or any saved examples to provide tailored code completions. PostBot, automatically generates code proposal for a specific test function based on its name/title, the up-to-date response, and all saved response examples. Everyone can save time as it suggests code completions as developers type. Recommends data-driven tests based on observed response patterns to ensure comprehensive test coverage.

Figure 7: Smart Suggestions

Intuitive Prompts and Improved Performance

Significant advancements in natural language prompts have made interactions with the tool more intuitive. Users can now input specific requests, such as generating custom visualizations or fixing broken tests, to achieve more precise outcomes. Moreover, the latest version of PostBot boasts a performance boost of over two times its previous speed, ensuring a smooth integration into the API development process. PostBot enables users to craft clear and concise prompts using natural language.

Figure 8: Sample Prompts

Expert Debugging

Debugging requests with Postbot: If you get an unexpected error when you send a request, you can ask Postbot for help. Select What's wrong? in the error message. Postbot will tell you about any problems it can identify, and it will offer possible solutions for fixing the issue. Postbot will scan your request for invalid characters or whitespace in the method, URL, parameters, headers, and body. Any identified issues will be highlighted to prevent potential errors. Can Refer to this link for more details: - https://learning.postman.com/docs/sending-requests/response-data/troubleshooting-api-requests/#debugging-requests-with-postbot/

Visualized Response

Powerful data visualization: Say goodbye to endless JSON scrolling! Postbot's data visualization magic brings your API responses to life. With a simple natural language request, you can transform your data into insightful tables, charts, or bar graphs, tailored to your specific needs. (e.g. can change the color of the bar, can customize to generate only for specific columns to display in the dashboard)

Figure 9: Visualization using table and line chart

Generate tests for collections

Streamlined API testing: Postbot will be able to create or update test suites through the Collection Runner. This is a time-saving feature, especially when dealing with large test suites. Automated test suite generation ensures comprehensive API testing. Postbot can generate tests for individual requests or entire collections, using natural language autocomplete to speed up the process. Tired of manually writing test scripts? Our "Generate tests" feature is here to save the day! With a single click, you can automatically generate test scripts for your entire collection, especially useful for collections built from OpenAI Specs or captured traffic. This ensures comprehensive API validation and accelerates your testing process.

Figure 10: Creation of Test Suite Collections

API Documentation

Instant API documentation: Postbot generates comprehensive API documentation with a single click, allowing to save hours of manual work and effortlessly keep API documentation up to date. Can request Postbot to generate documentation according to custom requirements, which ensures it meets any specific needs.

Figure 11: Sample API Documentations

Get Help

AI-powered support: When you need help, Postbot is there for you. With expertly curated instructions drawn from multiple sources, Postbot provides instant support to keep you in the flow. No more searching online for answers; just ask Postbot and get the guidance you need to move forward.

Postbot is trained on Postman’s Learning Center documentation. Can Query for any postman related questions to Postbot for help. This acts as IN-APP Assistance, Look for a "Help" or "Support" section within the Postman app. This often contains FAQs, tutorials, and troubleshooting guides, Postbot checks for Postman-related forums or discussion boards, refers to the official Postman documentation for detailed information on using Postbot and other features and explore blog posts and articles written by Postman users who share their experiences and tips.

Figure 12: GET SUPPORT from POSTBOT

POSTBOT Benefits & Challenges

Success Metrics to Evaluate Any API Testing AI tool

Evaluating the success of API Testing AI tool involves tracking various performance and usage metrics. Here are some key metrics shall consider:

Conclusion

The generative AI capabilities have the potential to elevate the way developers work with APIs, bringing forth a new era of seamless API testing and automation. POSTBOT AI, is Postman’s foray into Gen AI. Postman Automation ‘s commitment to delivering practical, valuable solutions ensures that generative AI will drive real-world workflows, revolutionizing the API development and testing landscape for years to come.

This is just the beginning! Postman's new generative AI model is poised to revolutionize the way we approach API security testing and automation. It's not just a gimmick; it's a game-changer. By harnessing the power of AI, we can become more efficient, effective, and intelligent API testers.

References

To compile this whitepaper, we delved into a wealth of information from reputable sources, including research papers and articles: