Credo AI Connect

Credo AI Connect is the library to interact with the Credo AI Governance Platform. It is responsible for receiving “evidence requirements” and sending “evidence”

In this demo we train a model on some made-up data, and send the results to the Credo AI Governance Platform. For a tool to help you assess your AI systems, see Credo AI Lens.

Setup

Connect installation instruction can be found on readthedocs

Find the code

Click here to download this notebook.

Useful keywords

Credo AI Platform: Also referred to as simply “Platform”. The central AI governance/reporting Platform, found at https://app.credo.ai/
credoconfig: configuration file to be copied in the user’s home folder
use_case_name: The name of your Use Case as it is registered on Credo AI Platform
policy_pack: A set of governance controls that a Use Case needs to satisfy. A Use Case can have multiple policy packs applied to it.
policy_pack_key: A unique identifier for a policy pack (where do we get this?)
assessment_plan_url: The link to the assessment plan, this is generated in the Platform and used to download the assessment plan in the Governance object. See example below.
evidence: Any evaluation of an AI system, formatted specifically to be uploaded to the platform.

Setup API Connection with the Platform

Get a config file

This file contains all the necessary information to connect Lens to the Credo AI Platform.

To generate the config file, once you logged into the platform, click on your name icon (top left) and follow:

My Settings -> Tokens -> Plus Sign -> Generate

Immediately after generating the token, you will be given the possibility to download the config file.

The default location/file name Lens is expecting is ~/.credoconfig, where ~ is your home folder. You can specify any other location when you are initializing the Governance object (see below).

Do some AI stuff!

[ ]:

%load_ext autoreload
%autoreload 2
from sklearn.linear_model import LinearRegression
import sklearn.metrics as sk_metrics
from sklearn.model_selection import train_test_split
import numpy as np

Get data and train model

We’ll make up some data and fit a simple model to get us started.

[ ]:

model = LinearRegression()
r = np.random.RandomState(42)

# hallucinate data
N = 10000
data = r.randn(N, 2)
y = (data[:,0] + 3*data[:,1] + r.randn(N)*.2)
# split
X_train, X_test, y_train, y_test = train_test_split(data, y)

# train model
model.fit(X_train, y_train)
pred_test = model.predict(X_test)

Assess the AI System

The most likely thing you’ll want to send to the Credo AI Platform are assessments of the AI system. These are often metrics that are calculated to summarize aspects of the system’s behavior (e.g., performance, fairness) but can be other things too, like descriptive statistics of the dataset.

We’ll assume we have a suite of assessments we plan to use for this use-case. The below is just an example - any assessments can be done and sent to the Platform.

[ ]:

required_assessments = [sk_metrics.r2_score, sk_metrics.mean_squared_error]

assessments = {m.__name__: m(y_test, pred_test) for m in required_assessments}

Platform integration in 5 Minutes

To send evidence to the platform takes three steps:

Connect to the platform via the Governance class
Use the Adapter class to send the assessments to the Governance class
Export the evidence to the platform.

First we will see all the code together, and then break it down.

[ ]:

import connect as ct

[ ]:

# connect to platform via governace
gov = ct.Governance()  # Specify config_path if your config file is not the default one: ~/.credoconfig
url = 'your assessment url'
gov.register(assessment_plan_url=url)

# set up adapter and send metrics to governance class
adapter = ct.Adapter(governance = gov, model_name='My Model')
source = f"Quickstart_Connect-{ct.__version__}"
adapter.metrics_to_governance(metrics=assessments, source = source)

# export
gov.export()

1. Connect to the platform via the Governance Class

The Governance class handles the connection with the Credo AI Platform. On the Platform, you can govern an AI system by specifying “policy packs” that specify the technical requirements the AI system must meet. This class can retrieve them, which is most useful if you are using Credo AI Lens, our assessment framework. If you aren’t using Lens, the requirements are still important in directing your assessment, but they aren’t programatically connected to your assessments.

Since we are only using Connect in this demo, we will ignore that functionality.

The important functionality relevant here is that the Governance object handles the API calls allowing you to send evidence to the platform.

[ ]:

# Retrieve Policy Pack Assessment Plan
gov = ct.Governance()  # Specify config_path if your config file is not the default one: ~/.credoconfig
url = 'your assessment url'
gov.register(assessment_plan_url=url)

2. Use the Adapter class to send the assessments to the Governance class

The Adapter class handles structuring your assessments so that the Credo AI Platform can understand them. Connect uses EvidenceContainers to handle converting python objects like dictionaries into Evidence, the structured output that the Platform can understand. The Adapter then passes this Evidence to the Governance object for export

[ ]:

adapter = ct.Adapter(governance = gov, model_name='My Model')

Once we initialize an adapter, we can use its functionality to send different kinds of evidence.

For instance, metrics (which must be organized as key:value pairs of metric_name:value) are sent to governance using metrics_to_governance This converts the dictionary of assessments into Evidence (in this case a MetricEvidence).

Every time we send evidence we specify a source. This is for you to define however you would like. We suggest you make the source useful to establish provenance of your assessments.

[ ]:

source = f"Quickstart_Connect-{ct.__version__}"
adapter.metrics_to_governance(metrics=assessments, source = source)

You can see the evidence in the Governance class we instantiated before

[ ]:

gov.get_evidence()

If you need to send more evidence, you’ll just call a function like metrics_to_governance again. By default, this functions overwright the Evidence in Governance so you’ll have to change an argument to allow overwriting.

We will send a different type of evidence - a table, which must be a pandas DataFrame.

[ ]:

import pandas as pd
table = pd.DataFrame({'arbitrary_data': [3,4,5]})
table.name = 'my_table'
adapter.table_to_governance(table, source=source, overwrite_governance=False)

[ ]:

gov.get_evidence()

3. Export the evidence to the platform.

Exporting is straight forward! You can either export directly to the Platform or to a file. At the time of export, the uploaded evidence will be checked against the governance requirements specified on Platform and let you know what’s missing.

[ ]:

# export to API
gov.export()

# export to file
gov.export("assessment_export.json")

Other ways to label your evidence

On the Credo AI Platform, certain governance requirements may only apply to models that are tagged in a certain way. You can get the requirements tags from governance and apply it by passing a dictionary to the model_tags of the Adapter

[ ]:

gov.get_requirement_tags()

[ ]:

adapter = ct.Adapter(governance = gov, model_name='My Model', model_tags={'model_type': 'regression'})

If you need to label your evidence with more information than just source you can do that too.