How to manage generative AI proof of concept projects
November 27, 2023
,
Markku Räsänen
Explore key strategies for managing Generative AI proof of concept projects, focusing on Large Language Models (LLMs).

Generative AI is going to transform many tasks and processes in organizations because it increases efficiency. That is why most organizations are looking at specific ways they can utilize generative AI and specifically large language models (LLM) in their business. These business-specific use cases are often the ones that produce the most value. These cases interact with the business data of the organization and require specific care in terms of managing the proof-of-concept project (POC). In this blog post we will go through some aspects of a successful LLM POC project.

Have clear POC objectives

Each POC project should have a clear objective. Usually, these objectives come from your internal business goals. If you are part of a team facilitating a POC for another team, it makes sense to also check that their business goals are fulfilled. For example, if the goal is to test how LLMs can help in the debugging process for industrial equipment, it makes sense to clarify what constitutes success.  

It is good practice to not aim for perfection in the first phases of trying out generative AI. For example, if an internal document review process currently takes around 4 hours. So, companies aim to automate the whole process. Instead, a good scope of a POC would be to automate 80-90% of the work with LLMs while still leaving humans to review and verify the results. This also overcomes the current limitations that most LLM models have which, is that they are not 100% accurate and may hallucinate meaning that they may come up with information that is not correct or does not exist.

Some POC objective examples:
  • Test out LLM technology for our tendering process
    • Investigate the process speedup using LLMs
    • Investigate the accuracy of LLMs for the use case
    • Investigate the costs of deploying LLMs for the use case in production
  • Deploy a safe code co-pilot system for our internal confidential source code
    • Investigate the increase in software engineering productivity
    • Investigate the required level of data security for production deployment
    • Investigate the costs of deploying the LLM use case versus productivity gain in production
A screenshot of a computerDescription automatically generated

Select the right POC use cases

Selecting the right use case is critical. Also having some level of understanding for the limitations of the LLM technology is critical in choosing the right use cases. Another important aspect is thinking what type of business information will be exposed in the process. Is extremely critical information involved? This then warrants the use of special systems and safeguards for overall data safety.  

It is good practice to collect 3-5 use cases from different business units, evaluate them and find whether there are synergies and similarities. It is also good to select impactful, but not the hardest to implement case as the first POC. This way, your organization will get an overall picture of what LLMs can do for your business, without overly complicating the POC.

Overall, good POC use cases are ones that are in your business areas that your team knows well. That way they can comment on the potential impact, and you can rank POCs based on their predicted impact.

Select your POC team members

Decide who needs to be involved. Usually, someone from the business team needs to be part of the team as well as someone who will technically implement the POC. Bigger organizations may also require that the data team is present. You should have clear roles and responsibilities.

It is good practice for a successful POC to make sure that there are stakeholders from each team involved. It is also crucial that there is commitment from the internal customer team to test and provide feedback during the POC testing period.

Sample POC team structure
  • POC owner
    • Someone from the relevant business unit
  • POC implementer
    • Someone from the data / AI team or equivalent
  • POC support
    • Someone from the IT team as needed
    • Someone from operations / legal as needed

Implement the POC correctly

When you are implementing the POC, decide where it will be implemented. Will you use production data or just synthetic, or purposely created data for the POC? The overall implementation process can also be managed by using tools such as ConfidentialMind, which will help you deploy the right resources to any cloud you are using, manage the data sources as well as deploy the applications themselves. ConfidentialMind will also help you manage access to the POC application if it is important for you to guard access to it. It can help you manage your internal IAM roles in the application deployment process.

It is good practice to collect user feedback as early and as much as possible. As generative AI is a rather new genre of technology, don’t be afraid to test partly incomplete solutions to better meet the user requirements and see how the technology fits into your organization and your plans. A common pitfall of POC implementation is that not enough feedback is collected and as such it is not clear if the POC succeeded or not.

Evaluate the results

Once the POC is complete it is important to evaluate the results. What went right and what went wrong? You should evaluate the POC results against the KPIs you chose such as accuracy, fit for business purpose, costs efficiency, time saving and so on.

Common POC KPIs include
  • Efficiency: time saving in percentages or hours/minutes
  • Cost saving: how much money was saved or how much more efficient the POC process is compared to the existing process. This can also include things like freeing up team member time to do other more high value tasks
  • New feature: how the POC contributes as a new feature to an existing product or service

Decide how to go from proof of concept to production

After your evaluation, it is time to proceed to production. Going into production with LLMs is a big step. The LLM technology is ever evolving, and models change all the time. For a successful production deployment, you should make sure that your application architecture is such that you can change the underlying models as needed or as they develop. Tools such as ConfidentialMind can help in this process.  

ConfidentialMind allows you to control all the aspects of your deployment such as changing the underlying LLM for the deployed application with a push of a button. This is helpful as it might also be that you would like to use a different kind of LLM for production due to security or cost concerns.  

If you would like to know how ConfidentialMind can help you manage your LLM POCs and deployments, don’t hesitate to contact us at info@confidentialmind.com.  

By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
info@confidentialmind.com
ConfidentialMind Head Office
Otakaari 7,
02150 Espoo, Finland