Compare commits
2 Commits
main
...
rohanoai/u
| Author | SHA1 | Date | |
|---|---|---|---|
|
34737747cd | ||
|
|
05c3558e2b |
5
.gitignore
vendored
5
.gitignore
vendored
@ -127,8 +127,3 @@ dmypy.json
|
||||
|
||||
# Pyre type checker
|
||||
.pyre/
|
||||
|
||||
# Data
|
||||
*transactions*.jsonl
|
||||
/examples/data/transactions*
|
||||
*.DS_Store
|
||||
|
||||
79
README.md
79
README.md
@ -1,8 +1,8 @@
|
||||
# OpenAI Cookbook
|
||||
|
||||
This repository shares example code and example prompts for accomplishing common tasks with the [OpenAI API].
|
||||
This repository shares example code and example prompts for accomplishing common tasks with the [OpenAI API](https://openai.com/api/).
|
||||
|
||||
To try these examples yourself, you’ll need an OpenAI account. [Create a free account to get started.][API Signup]
|
||||
To try these examples yourself, you’ll need an OpenAI account. [Create a free account to get started.](https://beta.openai.com/signup)
|
||||
|
||||
Most code examples are written in Python, though the concepts can be applied in any language.
|
||||
|
||||
@ -10,13 +10,13 @@ In the same way that a cookbook's recipes don't span all possible meals or techn
|
||||
|
||||
## Related resources
|
||||
|
||||
Beyond the code examples here, you can also learn about the [OpenAI API] from the following resources:
|
||||
Beyond the code examples here, you can also learn about the [OpenAI API](https://openai.com/api/) from the following resources:
|
||||
|
||||
* Try out GPT-3 in the [OpenAI Playground]
|
||||
* Read about the API in the [OpenAI Documentation]
|
||||
* Discuss the API in the [OpenAI Community Forum]
|
||||
* Look for help in the [OpenAI Help Center]
|
||||
* See example prompts in the [OpenAI Examples]
|
||||
* Try out GPT-3 in the [OpenAI Playground](https://beta.openai.com/playground)
|
||||
* Read about the API in the [OpenAI Documentation](https://beta.openai.com/docs/introduction)
|
||||
* Discuss the API in the [OpenAI Community Forum](https://community.openai.com/top?period=monthly)
|
||||
* Look for help in the [OpenAI Help Center](https://help.openai.com/en/)
|
||||
* See example prompts in the [OpenAI Examples](https://beta.openai.com/examples)
|
||||
|
||||
## Examples, organized by capability
|
||||
|
||||
@ -87,7 +87,7 @@ Beyond the code examples here, you can also learn about the [OpenAI API] from th
|
||||
|
||||
## How large language models work
|
||||
|
||||
[Large language models][Large language models Blog Post] are functions that map text to text. Given an input string of text, a large language model tries to predict the text that will come next.
|
||||
[Large language models](https://openai.com/blog/better-language-models/) are functions that map text to text. Given an input string of text, a large language model tries to predict the text that will come next.
|
||||
|
||||
The magic of large language models is that by being trained to minimize this prediction error over vast quantities of text, the models end up learning concepts useful for these predictions. For example, they learn concepts like:
|
||||
|
||||
@ -102,7 +102,7 @@ The magic of large language models is that by being trained to minimize this pre
|
||||
|
||||
None of these capabilities are explicitly programmed in - they all emerge as a result of training.
|
||||
|
||||
GPT-3's capabilities now power [hundreds of different software products][GPT3 Apps Blog Post], including productivity apps, education apps, games, and more.
|
||||
GPT-3's capabilities now power [hundreds of different software products](https://openai.com/blog/gpt-3-apps/), including productivity apps, education apps, games, and more.
|
||||
|
||||
## How to control a large language model
|
||||
|
||||
@ -120,7 +120,7 @@ An example of each is shown below.
|
||||
|
||||
### Instruction prompts
|
||||
|
||||
Instruction-following models (e.g., `text-davinci-003` or any model beginning with `text-`) are specially designed to follow instructions. Write your instruction at the top of the prompt (or at the bottom, or both), and the model will do its best to follow the instruction and then stop. Instructions can be detailed, so don't be afraid to write a paragraph explicitly detailing the output you want.
|
||||
Instruction-following models (e.g., `text-davinci-002` or any model beginning with `text-`) are specially designed to follow instructions. Write your instruction at the top of the prompt (or at the bottom, or both), and the model will do its best to follow the instruction and then stop. Instructions can be detailed, so don't be afraid to write a paragraph explicitly detailing the output you want.
|
||||
|
||||
Example instruction prompt:
|
||||
|
||||
@ -182,7 +182,7 @@ Output:
|
||||
|
||||
### Fine-tuned prompt example
|
||||
|
||||
With enough training examples, you can [fine-tune][Fine Tuning Docs] a custom model. In this case, instructions become unnecessary, as the model can learn the task from the training data provided. However, it can be helpful to include separator sequences (e.g., `->` or `###` or any string that doesn't commonly appear in your inputs) to tell the model when the prompt has ended and the output should begin. Without separator sequences, there is a risk that the model continues elaborating on the input text rather than starting on the answer you want to see.
|
||||
With enough training examples, you can [fine-tune](https://beta.openai.com/docs/guides/fine-tuning) a custom model. In this case, instructions become unnecessary, as the model can learn the task from the training data provided. However, it can be helpful to include separator sequences (e.g., `->` or `###` or any string that doesn't commonly appear in your inputs) to tell the model when the prompt has ended and the output should begin. Without separator sequences, there is a risk that the model continues elaborating on the input text rather than starting on the answer you want to see.
|
||||
|
||||
Example fine-tuned prompt (for a model that has been custom trained on similar prompt-completion pairs):
|
||||
|
||||
@ -203,7 +203,7 @@ Output:
|
||||
|
||||
### More prompt advice
|
||||
|
||||
For more prompt examples, visit [OpenAI Examples][OpenAI Examples].
|
||||
For more prompt examples, visit [OpenAI Examples](https://beta.openai.com/examples).
|
||||
|
||||
In general, the input prompt is the best lever for improving model outputs. You can try tricks like:
|
||||
|
||||
@ -253,7 +253,7 @@ In general, writing can work with any style of prompt. Experiment to see what wo
|
||||
|
||||
| | Advantages | Disadvantages |
|
||||
| ---------------------------------------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||||
| Instruction-following models<br>(e.g., `text-davinci-003`) | Easiest to use | Less creative; less diverse; harder to control tone, length, etc. |
|
||||
| Instruction-following models<br>(e.g., `text-davinci-002`) | Easiest to use | Less creative; less diverse; harder to control tone, length, etc. |
|
||||
| Base models<br>(e.g., `davinci`) | More creative | More expensive (as including examples demonstrations in prompt will cost tokens) |
|
||||
| Fine-tuned models | Can train off of many examples; cheaper than including examples in the prompt | Hard to gather training data; training makes iteration slower and more expensive |
|
||||
|
||||
@ -301,7 +301,7 @@ Output:
|
||||
One
|
||||
```
|
||||
|
||||
If the text you wish to ask about is longer than the token limit (~4,000 tokens for `text-davinci-003` and ~2,000 tokens for earlier models), we recommending splitting the text into smaller pieces, ranking them by relevance, and then asking the most-relevant-looking pieces.
|
||||
If the text you wish to ask about is longer than the token limit (~4,000 tokens for `text-davinci-002` and ~2,000 tokens for earlier models), we recommending splitting the text into smaller pieces, ranking them by relevance, and then asking the most-relevant-looking pieces.
|
||||
|
||||
#### Summarization
|
||||
|
||||
@ -360,7 +360,7 @@ People & titles: William Baxter (Assistant Attorney General), Charles L. Brown (
|
||||
|
||||
### 3. Edit text
|
||||
|
||||
In addition to the [completion API endpoint][Completion API Docs], OpenAI now offers an [edit API endpoint][Edit API Docs] ([blog post][GPT3 Edit Blog Post]). In contrast to completions, which only take a single text input, edits take two text inputs: the instruction and the text to be modified.
|
||||
In addition to the [completion API endpoint](https://beta.openai.com/docs/api-reference/completions), OpenAI now offers an [edit API endpoint](https://beta.openai.com/docs/api-reference/edits) ([blog post](https://openai.com/blog/gpt-3-edit-insert/)). In contrast to completions, which only take a single text input, edits take two text inputs: the instruction and the text to be modified.
|
||||
|
||||
An example edit prompt:
|
||||
|
||||
@ -435,7 +435,7 @@ When it comes to translation, large language models particularly shine at combin
|
||||
|
||||
### 4. Compare text
|
||||
|
||||
The [OpenAI API embeddings endpoint][Embeddings Docs] can be used to measure similarity between pieces of text ([blog post][Embeddings Blog Post]). By leveraging GPT-3's understanding of text, these embeddings [achieved state-of-the-art results](https://arxiv.org/abs/2201.10005) on benchmarks in both unsupervised learning and transfer learning settings.
|
||||
The [OpenAI API embeddings endpoint](https://beta.openai.com/docs/guides/embeddings) can be used to measure similarity between pieces of text ([blog post](https://openai.com/blog/introducing-text-and-code-embeddings/)). By leveraging GPT-3's understanding of text, these embeddings [achieved state-of-the-art results](https://arxiv.org/abs/2201.10005) on benchmarks in both unsupervised learning and transfer learning settings.
|
||||
|
||||
Embeddings can be used for semantic search, recommendations, cluster analysis, near-duplicate detection, and more.
|
||||
|
||||
@ -446,11 +446,11 @@ Embeddings can be used for search either by themselves or as a feature in a larg
|
||||
The simplest way to use embeddings for search is as follows:
|
||||
|
||||
* Before the search (precompute):
|
||||
* Split your text corpus into chunks smaller than the token limit (e.g., <8,000 tokens)
|
||||
* Embed each chunk
|
||||
* Split your text corpus into chunks smaller than the token limit (e.g., ~2,000 tokens)
|
||||
* Embed each chunk using a 'doc' model (e.g., `text-search-curie-doc-001`)
|
||||
* Store those embeddings in your own database or in a vector search provider like [Pinecone](https://www.pinecone.io) or [Weaviate](https://weaviate.io)
|
||||
* At the time of the search (live compute):
|
||||
* Embed the search query
|
||||
* Embed the search query using the correponding 'query' model (e.g. `text-search-curie-query-001`)
|
||||
* Find the closest embeddings in your database
|
||||
* Return the top results, ranked by cosine similarity
|
||||
|
||||
@ -460,7 +460,7 @@ In more advanced search systems, the the cosine similarity of embeddings can be
|
||||
|
||||
#### Recommendations
|
||||
|
||||
Recommendations are quite similar to search, except that instead of a free-form text query, the inputs are items in a set.
|
||||
Recommendations are quite similar to search, except that instead of a free-form text query, the inputs are items in a set. And instead of using pairs of doc-query models, you can use a single symmetric similarity model (e.g., `text-similarity-curie-001`).
|
||||
|
||||
An example of how to use embeddings for recommendations is shown in [Recommendation_using_embeddings.ipynb](examples/Recommendation_using_embeddings.ipynb).
|
||||
|
||||
@ -470,23 +470,23 @@ Similar to search, these cosine similarity scores can either be used on their ow
|
||||
|
||||
Although OpenAI's embedding model weights cannot be fine-tuned, you can still use training data to customize embeddings to your application.
|
||||
|
||||
In the following notebook, we provide an example method for customizing your embeddings using training data. The idea of the method is to train a custom matrix to multiply embedding vectors by in order to get new customized embeddings. With good training data, this custom matrix will highlight the features relevant to your training labels and suppress the rest. You can equivalently consider the matrix multiplication as (a) a modification of the embeddings or (b) a modification of the distance function used to measure the distances between embeddings.
|
||||
In the following notebook, we provide an example method for customizing your embeddings using training data. The idea of the method is to train a custom matrix to multiply embedding vectors by in order to get new customized embeddings. With good training data, this custom matrix will highlight the features relevant to your training labels and suppress the rest. You can equivalently consider the matrix mulitplication as (a) a modification of the embeddings or (b) a modification of the distance function used to measure the distances between embeddings.
|
||||
|
||||
* [Customizing_embeddings.ipynb](examples/Customizing_embeddings.ipynb)
|
||||
|
||||
## Code Capabilities
|
||||
|
||||
Large language models aren't only great at text - they can be great at code too. OpenAI's specialized code model is called [Codex].
|
||||
Large language models aren't only great at text - they can be great at code too. OpenAI's specialized code model is called [Codex](https://openai.com/blog/openai-codex/).
|
||||
|
||||
Codex powers [more than 70 products][Codex Apps Blog Post], including:
|
||||
Codex powers [more than 70 products](https://openai.com/blog/codex-apps/), including:
|
||||
|
||||
* [GitHub Copilot] (autocompletes code in VS Code and other IDEs)
|
||||
* [GitHub Copilot](https://copilot.github.com/) (autocompletes code in VS Code and other IDEs)
|
||||
* [Pygma](https://pygma.app/) (turns Figma designs into code)
|
||||
* [Replit](https://replit.com/) (has an 'Explain code' button and other features)
|
||||
* [Warp](https://www.warp.dev/) (a smart terminal with AI command search)
|
||||
* [Machinet](https://machinet.net/) (writes Java unit test templates)
|
||||
|
||||
Note that unlike instruction-following text models (e.g., `text-davinci-003`), Codex is *not* trained to follow instructions. As a result, designing good prompts can take more care.
|
||||
Note that unlike instruction-following text models (e.g., `text-davinci-002`), Codex is *not* trained to follow instructions. As a result, designing good prompts can take more care.
|
||||
|
||||
### 1. Write code
|
||||
|
||||
@ -523,7 +523,7 @@ Code explanation can be applied to many use cases:
|
||||
* Generating in-code documentation (e.g., Python docstrings, git commit messages)
|
||||
* Generating out-of-code documentation (e.g., man pages)
|
||||
* In an interactive code exploration tool
|
||||
* Communicating program results back to users via a natural language interface
|
||||
* Communicating program results back to users via a natural langauge interface
|
||||
|
||||
An example prompt for explaining code with `code-davinci-002`:
|
||||
|
||||
@ -618,9 +618,9 @@ As you can see, `code-davinci-edit-001` was able to successfully reduce the func
|
||||
|
||||
The OpenAI API also features code search embeddings, which can measure the relevance of a section of code to a text query, or the similarity between two sections of code.
|
||||
|
||||
OpenAI code search embeddings significantly improved the state-of-the-art on the [CodeSearchNet] evaluation suite, scoring 93.5% versus the previous record of 77.4%.
|
||||
OpenAI code search embeddings significantly improved the state-of-the-art on the [CodeSearchNet](https://github.com/github/CodeSearchNet) evaluation suite, scoring 93.5% versus the previous record of 77.4%.
|
||||
|
||||
Read more about OpenAI's code embeddings in the [blog post announcement][Embeddings Blog Post] or [documentation][Embeddings Docs].
|
||||
Read more about OpenAI's code embeddings in the [blog post announcement](https://openai.com/blog/introducing-text-and-code-embeddings/) or [documentation](https://beta.openai.com/docs/guides/embeddings).
|
||||
|
||||
Code embeddings can be useful for use cases such as:
|
||||
|
||||
@ -630,24 +630,3 @@ Code embeddings can be useful for use cases such as:
|
||||
An example of code search is shown in [Code_search.ipynb](examples/Code_search.ipynb).
|
||||
|
||||
We haven't written an example of code clustering, but the idea is the same as the text clustering in [Clustering.ipynb](examples/Clustering.ipynb).
|
||||
|
||||
|
||||
[OpenAI API]: https://openai.com/api/
|
||||
[Embeddings Docs]: https://beta.openai.com/docs/guides/embeddings
|
||||
[Edit API Docs]: https://beta.openai.com/docs/api-reference/edits
|
||||
[Completion API Docs]: https://beta.openai.com/docs/api-reference/completions
|
||||
[Fine Tuning Docs]: https://beta.openai.com/docs/guides/fine-tuning
|
||||
[CodeSearchNet]: https://github.com/github/CodeSearchNet
|
||||
[Embeddings Blog Post]: https://openai.com/blog/introducing-text-and-code-embeddings/
|
||||
[Codex Apps Blog Post]: https://openai.com/blog/codex-apps/
|
||||
[GPT3 Edit Blog Post]: https://openai.com/blog/gpt-3-edit-insert/
|
||||
[Large language models Blog Post]: https://openai.com/blog/better-language-models/
|
||||
[GitHub Copilot]: https://copilot.github.com/
|
||||
[Codex]: https://openai.com/blog/openai-codex/
|
||||
[API Signup]: https://beta.openai.com/signup
|
||||
[GPT3 Apps Blog Post]: https://openai.com/blog/gpt-3-apps/
|
||||
[OpenAI Playground]: https://beta.openai.com/playground
|
||||
[OpenAI Documentation]: https://beta.openai.com/docs/introduction
|
||||
[OpenAI Community Forum]: https://community.openai.com/top?period=monthly
|
||||
[OpenAI Help Center]: https://help.openai.com/en/
|
||||
[OpenAI Examples]: https://beta.openai.com/examples
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -1,13 +1,12 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Code search\n",
|
||||
"\n",
|
||||
"We index our own [openai-python code repository](https://github.com/openai/openai-python), and show how it can be searched. We implement a simple version of file parsing and extracting of functions from python files."
|
||||
"We index our own openai-python code repository, and show how it can be searched. We implement a simple version of file parsing and extracting of functions from python files."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -19,8 +18,8 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Total number of py files: 51\n",
|
||||
"Total number of functions extracted: 97\n"
|
||||
"Total number of py files: 40\n",
|
||||
"Total number of functions extracted: 64\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
@ -64,24 +63,25 @@
|
||||
"\n",
|
||||
"# get user root directory\n",
|
||||
"root_dir = os.path.expanduser(\"~\")\n",
|
||||
"# note: for this code to work, the openai-python repo must be downloaded and placed in your root directory\n",
|
||||
"\n",
|
||||
"# path to code repository directory\n",
|
||||
"code_root = root_dir + \"/openai-python\"\n",
|
||||
"\n",
|
||||
"code_files = [y for x in os.walk(code_root) for y in glob(os.path.join(x[0], '*.py'))]\n",
|
||||
"print(\"Total number of py files:\", len(code_files))\n",
|
||||
"\n",
|
||||
"if len(code_files) == 0:\n",
|
||||
" print(\"Double check that you have downloaded the openai-python repo and set the code_root variable correctly.\")\n",
|
||||
"\n",
|
||||
"all_funcs = []\n",
|
||||
"for code_file in code_files:\n",
|
||||
" funcs = list(get_functions(code_file))\n",
|
||||
" for func in funcs:\n",
|
||||
" all_funcs.append(func)\n",
|
||||
"\n",
|
||||
"print(\"Total number of functions extracted:\", len(all_funcs))"
|
||||
"print(\"Total number of functions extracted:\", len(all_funcs))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For code search models we use code-search-{model}-code to obtain embeddings for code snippets, and code-search-{model}-text to embed natural language queries."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -119,57 +119,64 @@
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>def _console_log_level():\\n if openai.log i...</td>\n",
|
||||
" <td>_console_log_level</td>\n",
|
||||
" <td>/openai/util.py</td>\n",
|
||||
" <td>[0.03389773145318031, -0.004390408284962177, 0...</td>\n",
|
||||
" <td>def semantic_search(engine, query, documents):...</td>\n",
|
||||
" <td>semantic_search</td>\n",
|
||||
" <td>/examples/semanticsearch/semanticsearch.py</td>\n",
|
||||
" <td>[-0.038976121693849564, -0.0031428150832653046...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>def log_debug(message, **params):\\n msg = l...</td>\n",
|
||||
" <td>log_debug</td>\n",
|
||||
" <td>/openai/util.py</td>\n",
|
||||
" <td>[-0.004034275189042091, 0.004895383026450872, ...</td>\n",
|
||||
" <td>def main():\\n parser = argparse.ArgumentPar...</td>\n",
|
||||
" <td>main</td>\n",
|
||||
" <td>/examples/semanticsearch/semanticsearch.py</td>\n",
|
||||
" <td>[-0.024289356544613838, -0.017748363316059113,...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>def log_info(message, **params):\\n msg = lo...</td>\n",
|
||||
" <td>log_info</td>\n",
|
||||
" <td>/openai/util.py</td>\n",
|
||||
" <td>[0.004882764536887407, 0.0033515947870910168, ...</td>\n",
|
||||
" <td>def get_candidates(\\n prompt: str,\\n sto...</td>\n",
|
||||
" <td>get_candidates</td>\n",
|
||||
" <td>/examples/codex/backtranslation.py</td>\n",
|
||||
" <td>[-0.04161201789975166, -0.0169310811907053, 0....</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>def log_warn(message, **params):\\n msg = lo...</td>\n",
|
||||
" <td>log_warn</td>\n",
|
||||
" <td>/openai/util.py</td>\n",
|
||||
" <td>[0.002535992069169879, -0.010829543694853783, ...</td>\n",
|
||||
" <td>def rindex(lst: List, value: str) -> int:\\n ...</td>\n",
|
||||
" <td>rindex</td>\n",
|
||||
" <td>/examples/codex/backtranslation.py</td>\n",
|
||||
" <td>[-0.027255680412054062, -0.007931121625006199,...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>def logfmt(props):\\n def fmt(key, val):\\n ...</td>\n",
|
||||
" <td>logfmt</td>\n",
|
||||
" <td>/openai/util.py</td>\n",
|
||||
" <td>[0.016732551157474518, 0.017367802560329437, 0...</td>\n",
|
||||
" <td>def eval_candidate(\\n candidate_answer: str...</td>\n",
|
||||
" <td>eval_candidate</td>\n",
|
||||
" <td>/examples/codex/backtranslation.py</td>\n",
|
||||
" <td>[-0.00999179296195507, -0.01640152558684349, 0...</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" code function_name \\\n",
|
||||
"0 def _console_log_level():\\n if openai.log i... _console_log_level \n",
|
||||
"1 def log_debug(message, **params):\\n msg = l... log_debug \n",
|
||||
"2 def log_info(message, **params):\\n msg = lo... log_info \n",
|
||||
"3 def log_warn(message, **params):\\n msg = lo... log_warn \n",
|
||||
"4 def logfmt(props):\\n def fmt(key, val):\\n ... logfmt \n",
|
||||
" code function_name \\\n",
|
||||
"0 def semantic_search(engine, query, documents):... semantic_search \n",
|
||||
"1 def main():\\n parser = argparse.ArgumentPar... main \n",
|
||||
"2 def get_candidates(\\n prompt: str,\\n sto... get_candidates \n",
|
||||
"3 def rindex(lst: List, value: str) -> int:\\n ... rindex \n",
|
||||
"4 def eval_candidate(\\n candidate_answer: str... eval_candidate \n",
|
||||
"\n",
|
||||
" filepath code_embedding \n",
|
||||
"0 /openai/util.py [0.03389773145318031, -0.004390408284962177, 0... \n",
|
||||
"1 /openai/util.py [-0.004034275189042091, 0.004895383026450872, ... \n",
|
||||
"2 /openai/util.py [0.004882764536887407, 0.0033515947870910168, ... \n",
|
||||
"3 /openai/util.py [0.002535992069169879, -0.010829543694853783, ... \n",
|
||||
"4 /openai/util.py [0.016732551157474518, 0.017367802560329437, 0... "
|
||||
" filepath \\\n",
|
||||
"0 /examples/semanticsearch/semanticsearch.py \n",
|
||||
"1 /examples/semanticsearch/semanticsearch.py \n",
|
||||
"2 /examples/codex/backtranslation.py \n",
|
||||
"3 /examples/codex/backtranslation.py \n",
|
||||
"4 /examples/codex/backtranslation.py \n",
|
||||
"\n",
|
||||
" code_embedding \n",
|
||||
"0 [-0.038976121693849564, -0.0031428150832653046... \n",
|
||||
"1 [-0.024289356544613838, -0.017748363316059113,... \n",
|
||||
"2 [-0.04161201789975166, -0.0169310811907053, 0.... \n",
|
||||
"3 [-0.027255680412054062, -0.007931121625006199,... \n",
|
||||
"4 [-0.00999179296195507, -0.01640152558684349, 0... "
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
@ -181,109 +188,12 @@
|
||||
"from openai.embeddings_utils import get_embedding\n",
|
||||
"\n",
|
||||
"df = pd.DataFrame(all_funcs)\n",
|
||||
"df['code_embedding'] = df['code'].apply(lambda x: get_embedding(x, engine='text-embedding-ada-002'))\n",
|
||||
"df['code_embedding'] = df['code'].apply(lambda x: get_embedding(x, engine='code-search-babbage-code-001'))\n",
|
||||
"df['filepath'] = df['filepath'].apply(lambda x: x.replace(code_root, \"\"))\n",
|
||||
"df.to_csv(\"data/code_search_openai-python.csv\", index=False)\n",
|
||||
"df.to_csv(\"output/code_search_openai-python.csv\", index=False)\n",
|
||||
"df.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/openai/tests/test_endpoints.py:test_completions score=0.826\n",
|
||||
"def test_completions():\n",
|
||||
" result = openai.Completion.create(prompt=\"This was a test\", n=5, engine=\"ada\")\n",
|
||||
" assert len(result.choices) == 5\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/tests/test_endpoints.py:test_completions_model score=0.811\n",
|
||||
"def test_completions_model():\n",
|
||||
" result = openai.Completion.create(prompt=\"This was a test\", n=5, model=\"ada\")\n",
|
||||
" assert len(result.choices) == 5\n",
|
||||
" assert result.model.startswith(\"ada\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/tests/test_endpoints.py:test_completions_multiple_prompts score=0.808\n",
|
||||
"def test_completions_multiple_prompts():\n",
|
||||
" result = openai.Completion.create(\n",
|
||||
" prompt=[\"This was a test\", \"This was another test\"], n=5, engine=\"ada\"\n",
|
||||
" )\n",
|
||||
" assert len(result.choices) == 10\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"----------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from openai.embeddings_utils import cosine_similarity\n",
|
||||
"\n",
|
||||
"def search_functions(df, code_query, n=3, pprint=True, n_lines=7):\n",
|
||||
" embedding = get_embedding(code_query, engine='text-embedding-ada-002')\n",
|
||||
" df['similarities'] = df.code_embedding.apply(lambda x: cosine_similarity(x, embedding))\n",
|
||||
"\n",
|
||||
" res = df.sort_values('similarities', ascending=False).head(n)\n",
|
||||
" if pprint:\n",
|
||||
" for r in res.iterrows():\n",
|
||||
" print(r[1].filepath+\":\"+r[1].function_name + \" score=\" + str(round(r[1].similarities, 3)))\n",
|
||||
" print(\"\\n\".join(r[1].code.split(\"\\n\")[:n_lines]))\n",
|
||||
" print('-'*70)\n",
|
||||
" return res\n",
|
||||
"\n",
|
||||
"res = search_functions(df, 'Completions API tests', n=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/openai/validators.py:format_inferrer_validator score=0.751\n",
|
||||
"def format_inferrer_validator(df):\n",
|
||||
" \"\"\"\n",
|
||||
" This validator will infer the likely fine-tuning format of the data, and display it to the user if it is classification.\n",
|
||||
" It will also suggest to use ada and explain train/validation split benefits.\n",
|
||||
" \"\"\"\n",
|
||||
" ft_type = infer_task_type(df)\n",
|
||||
" immediate_msg = None\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/validators.py:get_validators score=0.748\n",
|
||||
"def get_validators():\n",
|
||||
" return [\n",
|
||||
" num_examples_validator,\n",
|
||||
" lambda x: necessary_column_validator(x, \"prompt\"),\n",
|
||||
" lambda x: necessary_column_validator(x, \"completion\"),\n",
|
||||
" additional_column_validator,\n",
|
||||
" non_empty_field_validator,\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/validators.py:infer_task_type score=0.738\n",
|
||||
"def infer_task_type(df):\n",
|
||||
" \"\"\"\n",
|
||||
" Infer the likely fine-tuning task type from the data\n",
|
||||
" \"\"\"\n",
|
||||
" CLASSIFICATION_THRESHOLD = 3 # min_average instances of each class\n",
|
||||
" if sum(df.prompt.str.len()) == 0:\n",
|
||||
" return \"open-ended generation\"\n",
|
||||
"----------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_functions(df, 'fine-tuning input data validation logic', n=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
@ -293,35 +203,48 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/openai/validators.py:get_common_xfix score=0.793\n",
|
||||
"def get_common_xfix(series, xfix=\"suffix\"):\n",
|
||||
" \"\"\"\n",
|
||||
" Finds the longest common suffix or prefix of all the values in a series\n",
|
||||
" \"\"\"\n",
|
||||
" common_xfix = \"\"\n",
|
||||
" while True:\n",
|
||||
" common_xfixes = (\n",
|
||||
" series.str[-(len(common_xfix) + 1) :]\n",
|
||||
" if xfix == \"suffix\"\n",
|
||||
" else series.str[: len(common_xfix) + 1]\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/validators.py:common_completion_suffix_validator score=0.778\n",
|
||||
"def common_completion_suffix_validator(df):\n",
|
||||
" \"\"\"\n",
|
||||
" This validator will suggest to add a common suffix to the completion if one doesn't already exist in case of classification or conditional generation.\n",
|
||||
" \"\"\"\n",
|
||||
" error_msg = None\n",
|
||||
" immediate_msg = None\n",
|
||||
" optional_msg = None\n",
|
||||
" optional_fn = None\n",
|
||||
"/openai/tests/test_endpoints.py:test_completions_multiple_prompts score=0.681\n",
|
||||
"def test_completions_multiple_prompts():\n",
|
||||
" result = openai.Completion.create(\n",
|
||||
" prompt=[\"This was a test\", \"This was another test\"], n=5, engine=\"ada\"\n",
|
||||
" )\n",
|
||||
" assert len(result.choices) == 10\n",
|
||||
"\n",
|
||||
" ft_type = infer_task_type(df)\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/tests/test_endpoints.py:test_completions score=0.675\n",
|
||||
"def test_completions():\n",
|
||||
" result = openai.Completion.create(prompt=\"This was a test\", n=5, engine=\"ada\")\n",
|
||||
" assert len(result.choices) == 5\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/tests/test_api_requestor.py:test_requestor_sets_request_id score=0.635\n",
|
||||
"def test_requestor_sets_request_id(mocker: MockerFixture) -> None:\n",
|
||||
" # Fake out 'requests' and confirm that the X-Request-Id header is set.\n",
|
||||
"\n",
|
||||
" got_headers = {}\n",
|
||||
"\n",
|
||||
" def fake_request(self, *args, **kwargs):\n",
|
||||
" nonlocal got_headers\n",
|
||||
"----------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_functions(df, 'find common suffix', n=2, n_lines=10)"
|
||||
"from openai.embeddings_utils import cosine_similarity\n",
|
||||
"\n",
|
||||
"def search_functions(df, code_query, n=3, pprint=True, n_lines=7):\n",
|
||||
" embedding = get_embedding(code_query, engine='code-search-babbage-text-001')\n",
|
||||
" df['similarities'] = df.code_embedding.apply(lambda x: cosine_similarity(x, embedding))\n",
|
||||
"\n",
|
||||
" res = df.sort_values('similarities', ascending=False).head(n)\n",
|
||||
" if pprint:\n",
|
||||
" for r in res.iterrows():\n",
|
||||
" print(r[1].filepath+\":\"+r[1].function_name + \" score=\" + str(round(r[1].similarities, 3)))\n",
|
||||
" print(\"\\n\".join(r[1].code.split(\"\\n\")[:n_lines]))\n",
|
||||
" print('-'*70)\n",
|
||||
" return res\n",
|
||||
"res = search_functions(df, 'Completions API tests', n=3)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -333,7 +256,90 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/openai/cli.py:tools_register score=0.773\n",
|
||||
"/openai/validators.py:format_inferrer_validator score=0.655\n",
|
||||
"def format_inferrer_validator(df):\n",
|
||||
" \"\"\"\n",
|
||||
" This validator will infer the likely fine-tuning format of the data, and display it to the user if it is classification.\n",
|
||||
" It will also suggest to use ada and explain train/validation split benefits.\n",
|
||||
" \"\"\"\n",
|
||||
" ft_type = infer_task_type(df)\n",
|
||||
" immediate_msg = None\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/validators.py:long_examples_validator score=0.649\n",
|
||||
"def long_examples_validator(df):\n",
|
||||
" \"\"\"\n",
|
||||
" This validator will suggest to the user to remove examples that are too long.\n",
|
||||
" \"\"\"\n",
|
||||
" immediate_msg = None\n",
|
||||
" optional_msg = None\n",
|
||||
" optional_fn = None\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/validators.py:non_empty_completion_validator score=0.646\n",
|
||||
"def non_empty_completion_validator(df):\n",
|
||||
" \"\"\"\n",
|
||||
" This validator will ensure that no completion is empty.\n",
|
||||
" \"\"\"\n",
|
||||
" necessary_msg = None\n",
|
||||
" necessary_fn = None\n",
|
||||
" immediate_msg = None\n",
|
||||
"----------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_functions(df, 'fine-tuning input data validation logic', n=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/openai/validators.py:common_completion_suffix_validator score=0.665\n",
|
||||
"def common_completion_suffix_validator(df):\n",
|
||||
" \"\"\"\n",
|
||||
" This validator will suggest to add a common suffix to the completion if one doesn't already exist in case of classification or conditional generation.\n",
|
||||
" \"\"\"\n",
|
||||
" error_msg = None\n",
|
||||
" immediate_msg = None\n",
|
||||
" optional_msg = None\n",
|
||||
" optional_fn = None\n",
|
||||
"\n",
|
||||
" ft_type = infer_task_type(df)\n",
|
||||
"----------------------------------------------------------------------\n",
|
||||
"/openai/validators.py:get_outfnames score=0.66\n",
|
||||
"def get_outfnames(fname, split):\n",
|
||||
" suffixes = [\"_train\", \"_valid\"] if split else [\"\"]\n",
|
||||
" i = 0\n",
|
||||
" while True:\n",
|
||||
" index_suffix = f\" ({i})\" if i > 0 else \"\"\n",
|
||||
" candidate_fnames = [\n",
|
||||
" fname.split(\".\")[0] + \"_prepared\" + suffix + index_suffix + \".jsonl\"\n",
|
||||
" for suffix in suffixes\n",
|
||||
" ]\n",
|
||||
" if not any(os.path.isfile(f) for f in candidate_fnames):\n",
|
||||
"----------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_functions(df, 'find common suffix', n=2, n_lines=10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/openai/cli.py:tools_register score=0.651\n",
|
||||
"def tools_register(parser):\n",
|
||||
" subparsers = parser.add_subparsers(\n",
|
||||
" title=\"Tools\", help=\"Convenience client side tools\"\n",
|
||||
@ -368,9 +374,8 @@
|
||||
"hash": "be4b5d5b73a21c599de40d6deb1129796d12dc1cc33a738f7bac13269cfcafe8"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "openai-cookbook",
|
||||
"language": "python",
|
||||
"name": "openai-cookbook"
|
||||
"display_name": "Python 3.7.3 64-bit ('base': conda)",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
@ -382,7 +387,7 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.6"
|
||||
"version": "3.7.3"
|
||||
},
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
|
||||
@ -17,7 +17,7 @@
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"1536"
|
||||
"12288"
|
||||
]
|
||||
},
|
||||
"execution_count": 1,
|
||||
@ -29,8 +29,8 @@
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"embedding = openai.Embedding.create(\n",
|
||||
" input=\"Your text goes here\",\n",
|
||||
" engine=\"text-embedding-ada-002\"\n",
|
||||
" input=\"Sample document text goes here\",\n",
|
||||
" engine=\"text-similarity-davinci-001\"\n",
|
||||
")[\"data\"][0][\"embedding\"]\n",
|
||||
"len(embedding)\n"
|
||||
]
|
||||
@ -44,7 +44,7 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"1536\n"
|
||||
"1024\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
@ -54,7 +54,7 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6))\n",
|
||||
"def get_embedding(text: str, engine=\"text-embedding-ada-002\") -> list[float]:\n",
|
||||
"def get_embedding(text: str, engine=\"text-similarity-davinci-001\") -> list[float]:\n",
|
||||
"\n",
|
||||
" # replace newlines, which can negatively affect performance.\n",
|
||||
" text = text.replace(\"\\n\", \" \")\n",
|
||||
@ -62,7 +62,25 @@
|
||||
" return openai.Embedding.create(input=[text], engine=engine)[\"data\"][0][\"embedding\"]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"embedding = get_embedding(\"Your text goes here\", engine=\"text-embedding-ada-002\")\n",
|
||||
"embedding = get_embedding(\"Sample query text goes here\", engine=\"text-search-ada-query-001\")\n",
|
||||
"print(len(embedding))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"1024\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"embedding = get_embedding(\"Sample document text goes here\", engine=\"text-search-ada-doc-001\")\n",
|
||||
"print(len(embedding))\n"
|
||||
]
|
||||
}
|
||||
|
||||
@ -1,602 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# How to handle rate limits\n",
|
||||
"\n",
|
||||
"When you call the OpenAI API repeatedly, you may encounter error messages that say `429: 'Too Many Requests'` or `RateLimitError`. These error messages come from exceeding the API's rate limits.\n",
|
||||
"\n",
|
||||
"Rate limits are a common practice for APIs, and they're put in place for a few different reasons.\n",
|
||||
"\n",
|
||||
"- First, they help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, OpenAI can prevent this kind of activity.\n",
|
||||
"- Second, rate limits help ensure that everyone has fair access to the API. If one person or organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns.\n",
|
||||
"- Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users.\n",
|
||||
"\n",
|
||||
"Although hitting rate limits can be frustrating, rate limits exist to protect the reliable operation of the API for its users.\n",
|
||||
"\n",
|
||||
"In this guide, we'll share some tips for avoiding and handling rate limit errors."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Default rate limits\n",
|
||||
"\n",
|
||||
"As of Sep 2022, the default rate limits are:\n",
|
||||
"\n",
|
||||
"<table>\n",
|
||||
"<thead>\n",
|
||||
" <tr>\n",
|
||||
" <th></th>\n",
|
||||
" <th>Text Completion & Embedding endpoints</th>\n",
|
||||
" <th>Code & Edit endpoints</th>\n",
|
||||
" </tr>\n",
|
||||
"</thead>\n",
|
||||
"<tbody>\n",
|
||||
" <tr>\n",
|
||||
" <td>Free trial users</td>\n",
|
||||
" <td>\n",
|
||||
" <ul>\n",
|
||||
" <li>20 requests / minute</li>\n",
|
||||
" <li>150,000 tokens / minute</li>\n",
|
||||
" </ul>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <ul>\n",
|
||||
" <li>20 requests / minute</li>\n",
|
||||
" <li>150,000 tokens / minute</li>\n",
|
||||
" </ul>\n",
|
||||
" </td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>Pay-as-you-go users (in your first 48 hours)</td>\n",
|
||||
" <td>\n",
|
||||
" <ul>\n",
|
||||
" <li>60 requests / minute</li>\n",
|
||||
" <li>250,000 davinci tokens / minute (and proportionally more for smaller models)</li>\n",
|
||||
" </ul>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <ul>\n",
|
||||
" <li>20 requests / minute</li>\n",
|
||||
" <li>150,000 tokens / minute</li>\n",
|
||||
" </ul>\n",
|
||||
" </td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <td>Pay-as-you-go users (after your first 48 hours)</td>\n",
|
||||
" <td>\n",
|
||||
" <ul>\n",
|
||||
" <li>3,000 requests / minute</li>\n",
|
||||
" <li>250,000 davinci tokens / minute (and proportionally more for smaller models)</li>\n",
|
||||
" </ul>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <ul>\n",
|
||||
" <li>20 requests / minute</li>\n",
|
||||
" <li>150,000 tokens / minute</li>\n",
|
||||
" </ul>\n",
|
||||
" </td>\n",
|
||||
" </tr>\n",
|
||||
"</tbody>\n",
|
||||
"</table>\n",
|
||||
"\n",
|
||||
"For reference, 1,000 tokens is roughly a page of text.\n",
|
||||
"\n",
|
||||
"### Other rate limit resources\n",
|
||||
"\n",
|
||||
"Read more about OpenAI's rate limits in the [OpenAI Help Center](https://help.openai.com/en/):\n",
|
||||
"\n",
|
||||
"- [Is API usage subject to any rate limits?](https://help.openai.com/en/articles/5955598-is-api-usage-subject-to-any-rate-limits)\n",
|
||||
"- [How can I solve 429: 'Too Many Requests' errors?](https://help.openai.com/en/articles/5955604-how-can-i-solve-429-too-many-requests-errors)\n",
|
||||
"\n",
|
||||
"### Requesting a rate limit increase\n",
|
||||
"\n",
|
||||
"If you'd like your organization's rate limit increased, please fill out the following form:\n",
|
||||
"\n",
|
||||
"- [OpenAI Rate Limit Increase Request form](https://forms.gle/56ZrwXXoxAN1yt6i9)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example rate limit error\n",
|
||||
"\n",
|
||||
"A rate limit error will occur when API requests are sent too quickly. If using the OpenAI Python library, they will look something like:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you’d like to request an increase.\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Below is example code for triggering a rate limit error."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai # for making OpenAI API requests\n",
|
||||
"\n",
|
||||
"# request a bunch of completions in a loop\n",
|
||||
"for _ in range(100):\n",
|
||||
" openai.Completion.create(\n",
|
||||
" model=\"code-cushman-001\",\n",
|
||||
" prompt=\"def magic_function():\\n\\t\",\n",
|
||||
" max_tokens=10,\n",
|
||||
" )\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How to avoid rate limit errors\n",
|
||||
"\n",
|
||||
"### Retrying with exponential backoff\n",
|
||||
"\n",
|
||||
"One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.\n",
|
||||
"\n",
|
||||
"This approach has many benefits:\n",
|
||||
"\n",
|
||||
"- Automatic retries means you can recover from rate limit errors without crashes or missing data\n",
|
||||
"- Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail\n",
|
||||
"- Adding random jitter to the delay helps retries from all hitting at the same time\n",
|
||||
"\n",
|
||||
"Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.\n",
|
||||
"\n",
|
||||
"Below are a few example solutions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Example #1: Using the Tenacity library\n",
|
||||
"\n",
|
||||
"[Tenacity](https://tenacity.readthedocs.io/en/latest/) is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.\n",
|
||||
"\n",
|
||||
"To add exponential backoff to your requests, you can use the `tenacity.retry` [decorator](https://peps.python.org/pep-0318/). The following example uses the `tenacity.wait_random_exponential` function to add random exponential backoff to a request.\n",
|
||||
"\n",
|
||||
"Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<OpenAIObject text_completion id=cmpl-5oowO391reUW8RGVfFyzBM1uBs4A5 at 0x10d8cae00> JSON: {\n",
|
||||
" \"choices\": [\n",
|
||||
" {\n",
|
||||
" \"finish_reason\": \"length\",\n",
|
||||
" \"index\": 0,\n",
|
||||
" \"logprobs\": null,\n",
|
||||
" \"text\": \" a little girl dreamed of becoming a model.\\n\\nNowadays, that dream\"\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"created\": 1662793900,\n",
|
||||
" \"id\": \"cmpl-5oowO391reUW8RGVfFyzBM1uBs4A5\",\n",
|
||||
" \"model\": \"text-davinci-002\",\n",
|
||||
" \"object\": \"text_completion\",\n",
|
||||
" \"usage\": {\n",
|
||||
" \"completion_tokens\": 16,\n",
|
||||
" \"prompt_tokens\": 5,\n",
|
||||
" \"total_tokens\": 21\n",
|
||||
" }\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import openai # for OpenAI API calls\n",
|
||||
"from tenacity import (\n",
|
||||
" retry,\n",
|
||||
" stop_after_attempt,\n",
|
||||
" wait_random_exponential,\n",
|
||||
") # for exponential backoff\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))\n",
|
||||
"def completion_with_backoff(**kwargs):\n",
|
||||
" return openai.Completion.create(**kwargs)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"completion_with_backoff(model=\"text-davinci-002\", prompt=\"Once upon a time,\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Example #2: Using the backoff library\n",
|
||||
"\n",
|
||||
"Another library that provides function decorators for backoff and retry is [backoff](https://pypi.org/project/backoff/).\n",
|
||||
"\n",
|
||||
"Like Tenacity, the backoff library is a third-party tool, and OpenAI makes no guarantees about its reliability or security."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<OpenAIObject text_completion id=cmpl-5oowPhIdUvshEsF1rBhhwE9KFfI3M at 0x111043680> JSON: {\n",
|
||||
" \"choices\": [\n",
|
||||
" {\n",
|
||||
" \"finish_reason\": \"length\",\n",
|
||||
" \"index\": 0,\n",
|
||||
" \"logprobs\": null,\n",
|
||||
" \"text\": \" two children lived in a poor country village. In the winter, the temperature would\"\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"created\": 1662793901,\n",
|
||||
" \"id\": \"cmpl-5oowPhIdUvshEsF1rBhhwE9KFfI3M\",\n",
|
||||
" \"model\": \"text-davinci-002\",\n",
|
||||
" \"object\": \"text_completion\",\n",
|
||||
" \"usage\": {\n",
|
||||
" \"completion_tokens\": 16,\n",
|
||||
" \"prompt_tokens\": 5,\n",
|
||||
" \"total_tokens\": 21\n",
|
||||
" }\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import backoff # for exponential backoff\n",
|
||||
"import openai # for OpenAI API calls\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@backoff.on_exception(backoff.expo, openai.error.RateLimitError)\n",
|
||||
"def completions_with_backoff(**kwargs):\n",
|
||||
" return openai.Completion.create(**kwargs)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"completions_with_backoff(model=\"text-davinci-002\", prompt=\"Once upon a time,\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Example 3: Manual backoff implementation\n",
|
||||
"\n",
|
||||
"If you don't want to use third-party libraries, you can implement your own backoff logic."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<OpenAIObject text_completion id=cmpl-5oowRsCXv3AkUgVJyyo3TQrVq7hIT at 0x111024220> JSON: {\n",
|
||||
" \"choices\": [\n",
|
||||
" {\n",
|
||||
" \"finish_reason\": \"length\",\n",
|
||||
" \"index\": 0,\n",
|
||||
" \"logprobs\": null,\n",
|
||||
" \"text\": \" a man decided to greatly improve his karma by turning his life around.\\n\\n\"\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"created\": 1662793903,\n",
|
||||
" \"id\": \"cmpl-5oowRsCXv3AkUgVJyyo3TQrVq7hIT\",\n",
|
||||
" \"model\": \"text-davinci-002\",\n",
|
||||
" \"object\": \"text_completion\",\n",
|
||||
" \"usage\": {\n",
|
||||
" \"completion_tokens\": 16,\n",
|
||||
" \"prompt_tokens\": 5,\n",
|
||||
" \"total_tokens\": 21\n",
|
||||
" }\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# imports\n",
|
||||
"import random\n",
|
||||
"import time\n",
|
||||
"\n",
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"# define a retry decorator\n",
|
||||
"def retry_with_exponential_backoff(\n",
|
||||
" func,\n",
|
||||
" initial_delay: float = 1,\n",
|
||||
" exponential_base: float = 2,\n",
|
||||
" jitter: bool = True,\n",
|
||||
" max_retries: int = 10,\n",
|
||||
" errors: tuple = (openai.error.RateLimitError,),\n",
|
||||
"):\n",
|
||||
" \"\"\"Retry a function with exponential backoff.\"\"\"\n",
|
||||
"\n",
|
||||
" def wrapper(*args, **kwargs):\n",
|
||||
" # Initialize variables\n",
|
||||
" num_retries = 0\n",
|
||||
" delay = initial_delay\n",
|
||||
"\n",
|
||||
" # Loop until a successful response or max_retries is hit or an exception is raised\n",
|
||||
" while True:\n",
|
||||
" try:\n",
|
||||
" return func(*args, **kwargs)\n",
|
||||
"\n",
|
||||
" # Retry on specified errors\n",
|
||||
" except errors as e:\n",
|
||||
" # Increment retries\n",
|
||||
" num_retries += 1\n",
|
||||
"\n",
|
||||
" # Check if max retries has been reached\n",
|
||||
" if num_retries > max_retries:\n",
|
||||
" raise Exception(\n",
|
||||
" f\"Maximum number of retries ({max_retries}) exceeded.\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Increment the delay\n",
|
||||
" delay *= exponential_base * (1 + jitter * random.random())\n",
|
||||
"\n",
|
||||
" # Sleep for the delay\n",
|
||||
" time.sleep(delay)\n",
|
||||
"\n",
|
||||
" # Raise exceptions for any errors not specified\n",
|
||||
" except Exception as e:\n",
|
||||
" raise e\n",
|
||||
"\n",
|
||||
" return wrapper\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@retry_with_exponential_backoff\n",
|
||||
"def completions_with_backoff(**kwargs):\n",
|
||||
" return openai.Completion.create(**kwargs)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"completions_with_backoff(model=\"text-davinci-002\", prompt=\"Once upon a time,\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How to maximize throughput of batch processing given rate limits\n",
|
||||
"\n",
|
||||
"If you're processing real-time requests from users, backoff and retry is a great strategy to minimize latency while avoiding rate limit errors.\n",
|
||||
"\n",
|
||||
"However, if you're processing large volumes of batch data, where throughput matters more than latency, there are a few other things you can do in addition to backoff and retry.\n",
|
||||
"\n",
|
||||
"### Proactively adding delay between requests\n",
|
||||
"\n",
|
||||
"If you are constantly hitting the rate limit, then backing off, then hitting the rate limit again, then backing off again, it's possible that a good fraction of your request budget will be 'wasted' on requests that need to be retried. This limits your processing throughput, given a fixed rate limit.\n",
|
||||
"\n",
|
||||
"Here, one potential solution is to calculate your rate limit and add a delay equal to its reciprocal (e.g., if your rate limit 20 requests per minute, add a delay of 3 seconds to each request). This can help you operate near the rate limit ceiling without hitting it and incurring wasted requests.\n",
|
||||
"\n",
|
||||
"#### Example of adding delay to a request"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<OpenAIObject text_completion id=cmpl-5oowVVZnAzdCPtUJ0rifeamtLcZRp at 0x11b2c7680> JSON: {\n",
|
||||
" \"choices\": [\n",
|
||||
" {\n",
|
||||
" \"finish_reason\": \"length\",\n",
|
||||
" \"index\": 0,\n",
|
||||
" \"logprobs\": null,\n",
|
||||
" \"text\": \" there was an idyllic little farm that sat by a babbling brook\"\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"created\": 1662793907,\n",
|
||||
" \"id\": \"cmpl-5oowVVZnAzdCPtUJ0rifeamtLcZRp\",\n",
|
||||
" \"model\": \"text-davinci-002\",\n",
|
||||
" \"object\": \"text_completion\",\n",
|
||||
" \"usage\": {\n",
|
||||
" \"completion_tokens\": 16,\n",
|
||||
" \"prompt_tokens\": 5,\n",
|
||||
" \"total_tokens\": 21\n",
|
||||
" }\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# imports\n",
|
||||
"import time\n",
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"# Define a function that adds a delay to a Completion API call\n",
|
||||
"def delayed_completion(delay_in_seconds: float = 1, **kwargs):\n",
|
||||
" \"\"\"Delay a completion by a specified amount of time.\"\"\"\n",
|
||||
"\n",
|
||||
" # Sleep for the delay\n",
|
||||
" time.sleep(delay_in_seconds)\n",
|
||||
"\n",
|
||||
" # Call the Completion API and return the result\n",
|
||||
" return openai.Completion.create(**kwargs)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Calculate the delay based on your rate limit\n",
|
||||
"rate_limit_per_minute = 20\n",
|
||||
"delay = 60.0 / rate_limit_per_minute\n",
|
||||
"\n",
|
||||
"delayed_completion(\n",
|
||||
" delay_in_seconds=delay,\n",
|
||||
" model=\"text-davinci-002\",\n",
|
||||
" prompt=\"Once upon a time,\"\n",
|
||||
")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"\n",
|
||||
"### Batching requests\n",
|
||||
"\n",
|
||||
"The OpenAI API has separate limits for requests per minute and tokens per minute.\n",
|
||||
"\n",
|
||||
"If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with the smaller models.\n",
|
||||
"\n",
|
||||
"Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to `prompt` parameter instead of a single string.\n",
|
||||
"\n",
|
||||
"**Warning:** the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the `index` field.\n",
|
||||
"\n",
|
||||
"#### Example without batching"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Once upon a time, before there were grandiloquent tales of the massacre at Fort Mims, there were stories of\n",
|
||||
"Once upon a time, a full-sized search and rescue was created. However, CIDIs are the addition of requiring\n",
|
||||
"Once upon a time, Schubert was hot with the films. “Schubert sings of honey, flowers,\n",
|
||||
"Once upon a time, you could watch these films on your VCR, sometimes years after their initial theatrical release, and there\n",
|
||||
"Once upon a time, there was a forest. In that forest, the forest animals ruled. The forest animals had their homes\n",
|
||||
"Once upon a time, there were two programs that complained about false positive scans. Peacock and Midnight Manager alike, only\n",
|
||||
"Once upon a time, a long, long time ago, tragedy struck. it was the darkest of nights, and there was\n",
|
||||
"Once upon a time, when Adam was a perfect little gentleman, he was presented at Court as a guarantee of good character.\n",
|
||||
"Once upon a time, Adam and Eve made a mistake. They ate the fruit from the tree of immortality and split the consequences\n",
|
||||
"Once upon a time, there was a set of programming fundamental principles known as the \"X model.\" This is a set of\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import openai # for making OpenAI API requests\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"num_stories = 10\n",
|
||||
"prompt = \"Once upon a time,\"\n",
|
||||
"\n",
|
||||
"# serial example, with one story completion per request\n",
|
||||
"for _ in range(num_stories):\n",
|
||||
" response = openai.Completion.create(\n",
|
||||
" model=\"curie\",\n",
|
||||
" prompt=prompt,\n",
|
||||
" max_tokens=20,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # print story\n",
|
||||
" print(prompt + response.choices[0].text)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Example with batching"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Once upon a time, there were two sisters, Eliza Pickering and Ariana 'Ari' Lucas. When these lovely\n",
|
||||
"Once upon a time, Keene was stung by a worm — actually, probably a python — snaking through his leg\n",
|
||||
"Once upon a time, there was a professor of physics during the depression. It was difficult, during this time, to get\n",
|
||||
"Once upon a time, before you got sick, you told stories to all and sundry, and your listeners believed in you\n",
|
||||
"Once upon a time, there was one very old nice donkey. He was incredibly smart, in a very old, kind of\n",
|
||||
"Once upon a time, the property of a common lodging house was a common cup for all the inhabitants. Betimes a constant\n",
|
||||
"Once upon a time, in an unspecified country, there was a witch who had an illegal product. It was highly effective,\n",
|
||||
"Once upon a time, a long time ago, I turned 13, my beautiful dog Duncan swept me up into his jaws like\n",
|
||||
"Once upon a time, as a thoroughly reformed creature from an army of Nazis, he took On Judgement Day myself and his\n",
|
||||
"Once upon a time, Capcom made a game for the Atari VCS called Missile Command. While it was innovative at the time\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import openai # for making OpenAI API requests\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"num_stories = 10\n",
|
||||
"prompts = [\"Once upon a time,\"] * num_stories\n",
|
||||
"\n",
|
||||
"# batched example, with 10 stories completions per request\n",
|
||||
"response = openai.Completion.create(\n",
|
||||
" model=\"curie\",\n",
|
||||
" prompt=prompts,\n",
|
||||
" max_tokens=20,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# match completions to prompts by index\n",
|
||||
"stories = [\"\"] * len(prompts)\n",
|
||||
"for choice in response.choices:\n",
|
||||
" stories[choice.index] = prompts[choice.index] + choice.text\n",
|
||||
"\n",
|
||||
"# print stories\n",
|
||||
"for story in stories:\n",
|
||||
" print(story)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.9 ('openai')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.9"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@ -1,369 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# How to stream completions\n",
|
||||
"\n",
|
||||
"By default, when you send a prompt to the OpenAI Completions endpoint, it computes the entire completion and sends it back in a single response.\n",
|
||||
"\n",
|
||||
"If you're generating very long completions from a davinci-level model, waiting for the response can take many seconds. As of Aug 2022, responses from `text-davinci-002` typically take something like ~1 second plus ~2 seconds per 100 completion tokens.\n",
|
||||
"\n",
|
||||
"If you want to get the response faster, you can 'stream' the completion as it's being generated. This allows you to start printing or otherwise processing the beginning of the completion before the entire completion is finished.\n",
|
||||
"\n",
|
||||
"To stream completions, set `stream=True` when calling the Completions endpoint. This will return an object that streams back text as [data-only server-sent events](https://app.mode.com/openai/reports/4fce5ba22b5b/runs/f518a0be4495).\n",
|
||||
"\n",
|
||||
"Note that using `stream=True` in a production application makes it more difficult to moderate the content of the completions, which has implications for [approved usage](https://beta.openai.com/docs/usage-guidelines).\n",
|
||||
"\n",
|
||||
"Below is a Python code example of how to receive streaming completions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# imports\n",
|
||||
"import openai # for OpenAI API calls\n",
|
||||
"import time # for measuring time savings"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## A typical completion request\n",
|
||||
"\n",
|
||||
"With a typical Completions API call, the text is first computed and then returned all at once."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Full response received 7.32 seconds after request\n",
|
||||
"Full text received: 4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Example of an OpenAI Completion request\n",
|
||||
"# https://beta.openai.com/docs/api-reference/completions/create\n",
|
||||
"\n",
|
||||
"# record the time before the request is sent\n",
|
||||
"start_time = time.time()\n",
|
||||
"\n",
|
||||
"# send a Completion request to count to 100\n",
|
||||
"response = openai.Completion.create(\n",
|
||||
" model='text-davinci-002',\n",
|
||||
" prompt='1,2,3,',\n",
|
||||
" max_tokens=193,\n",
|
||||
" temperature=0,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# calculate the time it took to receive the response\n",
|
||||
"response_time = time.time() - start_time\n",
|
||||
"\n",
|
||||
"# extract the text from the response\n",
|
||||
"completion_text = response['choices'][0]['text']\n",
|
||||
"\n",
|
||||
"# print the time delay and text received\n",
|
||||
"print(f\"Full response received {response_time:.2f} seconds after request\")\n",
|
||||
"print(f\"Full text received: {completion_text}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## A streaming completion request\n",
|
||||
"\n",
|
||||
"With a streaming Completions API call, the text is sent back via a series of events. In Python, you can iterate over these events with a `for` loop."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Text received: 4 (0.16 seconds after request)\n",
|
||||
"Text received: , (0.19 seconds after request)\n",
|
||||
"Text received: 5 (0.21 seconds after request)\n",
|
||||
"Text received: , (0.24 seconds after request)\n",
|
||||
"Text received: 6 (0.27 seconds after request)\n",
|
||||
"Text received: , (0.29 seconds after request)\n",
|
||||
"Text received: 7 (0.32 seconds after request)\n",
|
||||
"Text received: , (0.35 seconds after request)\n",
|
||||
"Text received: 8 (0.37 seconds after request)\n",
|
||||
"Text received: , (0.40 seconds after request)\n",
|
||||
"Text received: 9 (0.43 seconds after request)\n",
|
||||
"Text received: , (0.46 seconds after request)\n",
|
||||
"Text received: 10 (0.48 seconds after request)\n",
|
||||
"Text received: , (0.51 seconds after request)\n",
|
||||
"Text received: 11 (0.54 seconds after request)\n",
|
||||
"Text received: , (0.56 seconds after request)\n",
|
||||
"Text received: 12 (0.59 seconds after request)\n",
|
||||
"Text received: , (0.62 seconds after request)\n",
|
||||
"Text received: 13 (0.64 seconds after request)\n",
|
||||
"Text received: , (0.67 seconds after request)\n",
|
||||
"Text received: 14 (0.70 seconds after request)\n",
|
||||
"Text received: , (0.72 seconds after request)\n",
|
||||
"Text received: 15 (0.75 seconds after request)\n",
|
||||
"Text received: , (0.78 seconds after request)\n",
|
||||
"Text received: 16 (0.84 seconds after request)\n",
|
||||
"Text received: , (0.84 seconds after request)\n",
|
||||
"Text received: 17 (0.86 seconds after request)\n",
|
||||
"Text received: , (0.89 seconds after request)\n",
|
||||
"Text received: 18 (0.91 seconds after request)\n",
|
||||
"Text received: , (0.94 seconds after request)\n",
|
||||
"Text received: 19 (1.41 seconds after request)\n",
|
||||
"Text received: , (1.41 seconds after request)\n",
|
||||
"Text received: 20 (1.41 seconds after request)\n",
|
||||
"Text received: , (1.41 seconds after request)\n",
|
||||
"Text received: 21 (1.41 seconds after request)\n",
|
||||
"Text received: , (1.41 seconds after request)\n",
|
||||
"Text received: 22 (1.41 seconds after request)\n",
|
||||
"Text received: , (1.41 seconds after request)\n",
|
||||
"Text received: 23 (1.41 seconds after request)\n",
|
||||
"Text received: , (1.41 seconds after request)\n",
|
||||
"Text received: 24 (1.46 seconds after request)\n",
|
||||
"Text received: , (1.46 seconds after request)\n",
|
||||
"Text received: 25 (1.46 seconds after request)\n",
|
||||
"Text received: , (1.55 seconds after request)\n",
|
||||
"Text received: 26 (1.61 seconds after request)\n",
|
||||
"Text received: , (1.65 seconds after request)\n",
|
||||
"Text received: 27 (1.66 seconds after request)\n",
|
||||
"Text received: , (1.70 seconds after request)\n",
|
||||
"Text received: 28 (1.72 seconds after request)\n",
|
||||
"Text received: , (1.75 seconds after request)\n",
|
||||
"Text received: 29 (1.78 seconds after request)\n",
|
||||
"Text received: , (2.05 seconds after request)\n",
|
||||
"Text received: 30 (2.08 seconds after request)\n",
|
||||
"Text received: , (2.13 seconds after request)\n",
|
||||
"Text received: 31 (2.16 seconds after request)\n",
|
||||
"Text received: , (2.20 seconds after request)\n",
|
||||
"Text received: 32 (2.26 seconds after request)\n",
|
||||
"Text received: , (2.28 seconds after request)\n",
|
||||
"Text received: 33 (2.31 seconds after request)\n",
|
||||
"Text received: , (2.35 seconds after request)\n",
|
||||
"Text received: 34 (2.38 seconds after request)\n",
|
||||
"Text received: , (2.54 seconds after request)\n",
|
||||
"Text received: 35 (2.55 seconds after request)\n",
|
||||
"Text received: , (2.59 seconds after request)\n",
|
||||
"Text received: 36 (2.61 seconds after request)\n",
|
||||
"Text received: , (2.64 seconds after request)\n",
|
||||
"Text received: 37 (2.67 seconds after request)\n",
|
||||
"Text received: , (2.71 seconds after request)\n",
|
||||
"Text received: 38 (2.86 seconds after request)\n",
|
||||
"Text received: , (2.89 seconds after request)\n",
|
||||
"Text received: 39 (2.92 seconds after request)\n",
|
||||
"Text received: , (2.95 seconds after request)\n",
|
||||
"Text received: 40 (2.99 seconds after request)\n",
|
||||
"Text received: , (3.01 seconds after request)\n",
|
||||
"Text received: 41 (3.04 seconds after request)\n",
|
||||
"Text received: , (3.08 seconds after request)\n",
|
||||
"Text received: 42 (3.15 seconds after request)\n",
|
||||
"Text received: , (3.33 seconds after request)\n",
|
||||
"Text received: 43 (3.36 seconds after request)\n",
|
||||
"Text received: , (3.43 seconds after request)\n",
|
||||
"Text received: 44 (3.47 seconds after request)\n",
|
||||
"Text received: , (3.50 seconds after request)\n",
|
||||
"Text received: 45 (3.53 seconds after request)\n",
|
||||
"Text received: , (3.56 seconds after request)\n",
|
||||
"Text received: 46 (3.59 seconds after request)\n",
|
||||
"Text received: , (3.63 seconds after request)\n",
|
||||
"Text received: 47 (3.65 seconds after request)\n",
|
||||
"Text received: , (3.68 seconds after request)\n",
|
||||
"Text received: 48 (3.71 seconds after request)\n",
|
||||
"Text received: , (3.77 seconds after request)\n",
|
||||
"Text received: 49 (3.77 seconds after request)\n",
|
||||
"Text received: , (3.79 seconds after request)\n",
|
||||
"Text received: 50 (3.82 seconds after request)\n",
|
||||
"Text received: , (3.85 seconds after request)\n",
|
||||
"Text received: 51 (3.89 seconds after request)\n",
|
||||
"Text received: , (3.91 seconds after request)\n",
|
||||
"Text received: 52 (3.93 seconds after request)\n",
|
||||
"Text received: , (3.96 seconds after request)\n",
|
||||
"Text received: 53 (3.98 seconds after request)\n",
|
||||
"Text received: , (4.04 seconds after request)\n",
|
||||
"Text received: 54 (4.05 seconds after request)\n",
|
||||
"Text received: , (4.07 seconds after request)\n",
|
||||
"Text received: 55 (4.10 seconds after request)\n",
|
||||
"Text received: , (4.13 seconds after request)\n",
|
||||
"Text received: 56 (4.19 seconds after request)\n",
|
||||
"Text received: , (4.20 seconds after request)\n",
|
||||
"Text received: 57 (4.20 seconds after request)\n",
|
||||
"Text received: , (4.23 seconds after request)\n",
|
||||
"Text received: 58 (4.26 seconds after request)\n",
|
||||
"Text received: , (4.30 seconds after request)\n",
|
||||
"Text received: 59 (4.31 seconds after request)\n",
|
||||
"Text received: , (4.59 seconds after request)\n",
|
||||
"Text received: 60 (4.61 seconds after request)\n",
|
||||
"Text received: , (4.64 seconds after request)\n",
|
||||
"Text received: 61 (4.67 seconds after request)\n",
|
||||
"Text received: , (4.72 seconds after request)\n",
|
||||
"Text received: 62 (4.73 seconds after request)\n",
|
||||
"Text received: , (4.76 seconds after request)\n",
|
||||
"Text received: 63 (4.80 seconds after request)\n",
|
||||
"Text received: , (4.83 seconds after request)\n",
|
||||
"Text received: 64 (4.86 seconds after request)\n",
|
||||
"Text received: , (4.89 seconds after request)\n",
|
||||
"Text received: 65 (4.92 seconds after request)\n",
|
||||
"Text received: , (4.94 seconds after request)\n",
|
||||
"Text received: 66 (4.97 seconds after request)\n",
|
||||
"Text received: , (5.00 seconds after request)\n",
|
||||
"Text received: 67 (5.03 seconds after request)\n",
|
||||
"Text received: , (5.06 seconds after request)\n",
|
||||
"Text received: 68 (5.09 seconds after request)\n",
|
||||
"Text received: , (5.14 seconds after request)\n",
|
||||
"Text received: 69 (5.16 seconds after request)\n",
|
||||
"Text received: , (5.19 seconds after request)\n",
|
||||
"Text received: 70 (5.22 seconds after request)\n",
|
||||
"Text received: , (5.28 seconds after request)\n",
|
||||
"Text received: 71 (5.30 seconds after request)\n",
|
||||
"Text received: , (5.33 seconds after request)\n",
|
||||
"Text received: 72 (5.36 seconds after request)\n",
|
||||
"Text received: , (5.38 seconds after request)\n",
|
||||
"Text received: 73 (5.41 seconds after request)\n",
|
||||
"Text received: , (5.44 seconds after request)\n",
|
||||
"Text received: 74 (5.48 seconds after request)\n",
|
||||
"Text received: , (5.51 seconds after request)\n",
|
||||
"Text received: 75 (5.53 seconds after request)\n",
|
||||
"Text received: , (5.56 seconds after request)\n",
|
||||
"Text received: 76 (5.60 seconds after request)\n",
|
||||
"Text received: , (5.62 seconds after request)\n",
|
||||
"Text received: 77 (5.65 seconds after request)\n",
|
||||
"Text received: , (5.68 seconds after request)\n",
|
||||
"Text received: 78 (5.71 seconds after request)\n",
|
||||
"Text received: , (5.77 seconds after request)\n",
|
||||
"Text received: 79 (5.77 seconds after request)\n",
|
||||
"Text received: , (5.79 seconds after request)\n",
|
||||
"Text received: 80 (5.82 seconds after request)\n",
|
||||
"Text received: , (5.85 seconds after request)\n",
|
||||
"Text received: 81 (5.88 seconds after request)\n",
|
||||
"Text received: , (5.92 seconds after request)\n",
|
||||
"Text received: 82 (5.93 seconds after request)\n",
|
||||
"Text received: , (5.97 seconds after request)\n",
|
||||
"Text received: 83 (5.98 seconds after request)\n",
|
||||
"Text received: , (6.01 seconds after request)\n",
|
||||
"Text received: 84 (6.04 seconds after request)\n",
|
||||
"Text received: , (6.07 seconds after request)\n",
|
||||
"Text received: 85 (6.09 seconds after request)\n",
|
||||
"Text received: , (6.11 seconds after request)\n",
|
||||
"Text received: 86 (6.14 seconds after request)\n",
|
||||
"Text received: , (6.17 seconds after request)\n",
|
||||
"Text received: 87 (6.19 seconds after request)\n",
|
||||
"Text received: , (6.22 seconds after request)\n",
|
||||
"Text received: 88 (6.24 seconds after request)\n",
|
||||
"Text received: , (6.27 seconds after request)\n",
|
||||
"Text received: 89 (6.30 seconds after request)\n",
|
||||
"Text received: , (6.31 seconds after request)\n",
|
||||
"Text received: 90 (6.35 seconds after request)\n",
|
||||
"Text received: , (6.36 seconds after request)\n",
|
||||
"Text received: 91 (6.40 seconds after request)\n",
|
||||
"Text received: , (6.44 seconds after request)\n",
|
||||
"Text received: 92 (6.46 seconds after request)\n",
|
||||
"Text received: , (6.49 seconds after request)\n",
|
||||
"Text received: 93 (6.51 seconds after request)\n",
|
||||
"Text received: , (6.54 seconds after request)\n",
|
||||
"Text received: 94 (6.56 seconds after request)\n",
|
||||
"Text received: , (6.59 seconds after request)\n",
|
||||
"Text received: 95 (6.62 seconds after request)\n",
|
||||
"Text received: , (6.64 seconds after request)\n",
|
||||
"Text received: 96 (6.68 seconds after request)\n",
|
||||
"Text received: , (6.68 seconds after request)\n",
|
||||
"Text received: 97 (6.70 seconds after request)\n",
|
||||
"Text received: , (6.73 seconds after request)\n",
|
||||
"Text received: 98 (6.75 seconds after request)\n",
|
||||
"Text received: , (6.78 seconds after request)\n",
|
||||
"Text received: 99 (6.90 seconds after request)\n",
|
||||
"Text received: , (6.92 seconds after request)\n",
|
||||
"Text received: 100 (7.25 seconds after request)\n",
|
||||
"Full response received 7.25 seconds after request\n",
|
||||
"Full text received: 4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Example of an OpenAI Completion request, using the stream=True option\n",
|
||||
"# https://beta.openai.com/docs/api-reference/completions/create\n",
|
||||
"\n",
|
||||
"# record the time before the request is sent\n",
|
||||
"start_time = time.time()\n",
|
||||
"\n",
|
||||
"# send a Completion request to count to 100\n",
|
||||
"response = openai.Completion.create(\n",
|
||||
" model='text-davinci-002',\n",
|
||||
" prompt='1,2,3,',\n",
|
||||
" max_tokens=193,\n",
|
||||
" temperature=0,\n",
|
||||
" stream=True, # this time, we set stream=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# create variables to collect the stream of events\n",
|
||||
"collected_events = []\n",
|
||||
"completion_text = ''\n",
|
||||
"# iterate through the stream of events\n",
|
||||
"for event in response:\n",
|
||||
" event_time = time.time() - start_time # calculate the time delay of the event\n",
|
||||
" collected_events.append(event) # save the event response\n",
|
||||
" event_text = event['choices'][0]['text'] # extract the text\n",
|
||||
" completion_text += event_text # append the text\n",
|
||||
" print(f\"Text received: {event_text} ({event_time:.2f} seconds after request)\") # print the delay and text\n",
|
||||
"\n",
|
||||
"# print the time delay and text received\n",
|
||||
"print(f\"Full response received {event_time:.2f} seconds after request\")\n",
|
||||
"print(f\"Full text received: {completion_text}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Time comparison\n",
|
||||
"\n",
|
||||
"In the example above, both requests took about 7 seconds to fully complete.\n",
|
||||
"\n",
|
||||
"However, with the streaming request, you would have received the first token after 0.16 seconds, and subsequent tokens after about ~0.035 seconds each."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.9 ('openai')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.9"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@ -11,14 +11,6 @@
|
||||
"We will combine the review summary and review text into a single combined text. The model will encode this combined text and it will output a single vector embedding."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To run this notebook, you will need to install: pandas, openai, transformers, plotly, matplotlib, scikit-learn, torch (transformer dep), torchvision, and scipy."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
@ -139,7 +131,7 @@
|
||||
"\n",
|
||||
"# remove reviews that are too long\n",
|
||||
"df['n_tokens'] = df.combined.apply(lambda x: len(tokenizer.encode(x)))\n",
|
||||
"df = df[df.n_tokens<8000].tail(1_000)\n",
|
||||
"df = df[df.n_tokens<2000].tail(1_000)\n",
|
||||
"len(df)"
|
||||
]
|
||||
},
|
||||
@ -156,22 +148,20 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"from openai.embeddings_utils import get_embedding\n",
|
||||
"# Ensure you have your API key set in your environment per the README: https://github.com/openai/openai-python#usage\n",
|
||||
"\n",
|
||||
"# This will take just between 5 and 10 minutes\n",
|
||||
"df['ada_similarity'] = df.combined.apply(lambda x: get_embedding(x, engine='text-embedding-ada-002'))\n",
|
||||
"df['ada_search'] = df['ada_similarity']\n",
|
||||
"# This will take just under 10 minutes\n",
|
||||
"df['babbage_similarity'] = df.combined.apply(lambda x: get_embedding(x, engine='text-similarity-babbage-001'))\n",
|
||||
"df['babbage_search'] = df.combined.apply(lambda x: get_embedding(x, engine='text-search-babbage-doc-001'))\n",
|
||||
"df.to_csv('data/fine_food_reviews_with_embeddings_1k.csv')"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "openai-cookbook",
|
||||
"display_name": "Python 3.9.9 ('openai')",
|
||||
"language": "python",
|
||||
"name": "openai-cookbook"
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
@ -183,12 +173,12 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.6"
|
||||
"version": "3.9.9"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
|
||||
"hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@ -195,7 +195,7 @@
|
||||
"\n",
|
||||
"We plan to use document embeddings to fetch the most relevant part of parts of our document library and insert them into the prompt that we provide to GPT-3. We therefore need to break up the document library into \"sections\" of context, which can be searched and retrieved separately. \n",
|
||||
"\n",
|
||||
"Sections should be large enough to contain enough information to answer a question; but small enough to fit one or several into the GPT-3 prompt. We find that approximately a paragraph of text is usually a good length, but you should experiment for your particular use case. In this example, Wikipedia articles are already grouped into semantically related headers, so we will use these to define our sections. This preprocessing has already been done in [this notebook](fine-tuned_qa/olympics-1-collect-data.ipynb), so we will load the results and use them."
|
||||
"Sections should be large enough to contain enough information to answer a question; but small enough to fit one or several into the GPT-3 prompt. We find that approximately a paragraph of text is usually a good length, but you should experiment for your particular use case. In this example, Wikipedia articles are already grouped into semantically related headers, so we will use these to define our sections. This preprocessing has already been done in [this notebook](examples/fine-tuned_qa/olympics-1-collect-data.ipynb), so we will load the results and use them."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -316,11 +316,11 @@
|
||||
"id": "a17b88b9-7ea2-491e-9727-12617c74a77d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We preprocess the document sections by creating an embedding vector for each section. An embedding is a vector of numbers that helps us understand how semantically similar or different the texts are. The closer two embeddings are to each other, the more similar are their contents. See the [documentation on OpenAI embeddings](https://beta.openai.com/docs/guides/embeddings) for more information.\n",
|
||||
"We preprocess the document sections by creating an embedding vector for each section. An embedding is a vector of numbers that helps us understand how semantically similar or different the texts are. The closer two embeddings are to each other, the more similar are their contents. See the [documentation on OpenAI embeddings](https://beta.api.openai.org/docs/guides/embeddings/) for more information.\n",
|
||||
"\n",
|
||||
"This indexing stage can be executed offline and only runs once to precompute the indexes for the dataset so that each piece of content can be retrieved later. Since this is a small example, we will store and search the embeddings locally. If you have a larger dataset, consider using a vector search engine like [Pinecone](https://www.pinecone.io/) or [Weaviate](https://github.com/semi-technologies/weaviate) to power the search.\n",
|
||||
"\n",
|
||||
"For the purposes of this tutorial we chose to use Curie embeddings, which are 4096-dimensional embeddings at a very good price and performance point. Since we will be using these embeddings for retrieval, we’ll use the \"search\" embeddings (see the [documentation](https://beta.openai.com/docs/guides/embeddings))."
|
||||
"For the purposes of this tutorial we chose to use Curie embeddings, which are 4096-dimensional embeddings at a very good price and performance point. Since we will be using these embeddings for retrieval, we’ll use the \"search\" embeddings (see the [documentation](https://beta.api.openai.org/docs/guides/embeddings/))."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -20,7 +20,7 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Ada similarity embedding performance on 1k Amazon reviews: mse=0.60, mae=0.51\n"
|
||||
"Babbage similarity embedding performance on 1k Amazon reviews: mse=0.39, mae=0.38\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
@ -32,13 +32,11 @@
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
|
||||
"\n",
|
||||
"# If you have not run the \"Obtain_dataset.ipynb\" notebook, you can download the datafile from here: https://cdn.openai.com/API/examples/data/fine_food_reviews_with_embeddings_1k.csv\n",
|
||||
"datafile_path = \"./data/fine_food_reviews_with_embeddings_1k.csv\"\n",
|
||||
"\n",
|
||||
"datafile_path = \"https://cdn.openai.com/API/examples/data/fine_food_reviews_with_embeddings_1k.csv\" # for your convenience, we precomputed the embeddings\n",
|
||||
"df = pd.read_csv(datafile_path)\n",
|
||||
"df[\"ada_similarity\"] = df.ada_similarity.apply(eval).apply(np.array)\n",
|
||||
"df[\"babbage_similarity\"] = df.babbage_similarity.apply(eval).apply(np.array)\n",
|
||||
"\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(list(df.ada_similarity.values), df.Score, test_size=0.2, random_state=42)\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(list(df.babbage_similarity.values), df.Score, test_size=0.2, random_state=42)\n",
|
||||
"\n",
|
||||
"rfr = RandomForestRegressor(n_estimators=100)\n",
|
||||
"rfr.fit(X_train, y_train)\n",
|
||||
@ -47,7 +45,7 @@
|
||||
"mse = mean_squared_error(y_test, preds)\n",
|
||||
"mae = mean_absolute_error(y_test, preds)\n",
|
||||
"\n",
|
||||
"print(f\"Ada similarity embedding performance on 1k Amazon reviews: mse={mse:.2f}, mae={mae:.2f}\")\n"
|
||||
"print(f\"Babbage similarity embedding performance on 1k Amazon reviews: mse={mse:.2f}, mae={mae:.2f}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -59,7 +57,7 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Dummy mean prediction performance on Amazon reviews: mse=1.73, mae=1.03\n"
|
||||
"Dummy mean prediction performance on Amazon reviews: mse=1.81, mae=1.08\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
@ -72,11 +70,10 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can see that the embeddings are able to predict the scores with an average error of 0.60 per score prediction. This is roughly equivalent to predicting 1 out of 3 reviews perfectly, and 1 out of two reviews by a one star error."
|
||||
"We can see that the embeddings are able to predict the scores with an average error of 0.39 per score prediction. This is roughly equivalent to predicting 2 out of 3 reviews perfectly, and 1 out of three reviews by a one star error."
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -89,9 +86,9 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "openai-cookbook",
|
||||
"display_name": "Python 3.9.9 ('openai')",
|
||||
"language": "python",
|
||||
"name": "openai-cookbook"
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
@ -103,7 +100,7 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.6"
|
||||
"version": "3.9.9"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
|
||||
@ -18,11 +18,9 @@
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"# If you have not run the \"Obtain_dataset.ipynb\" notebook, you can download the datafile from here: https://cdn.openai.com/API/examples/data/fine_food_reviews_with_embeddings_1k.csv\n",
|
||||
"datafile_path = \"./data/fine_food_reviews_with_embeddings_1k.csv\"\n",
|
||||
"\n",
|
||||
"datafile_path = \"https://cdn.openai.com/API/examples/data/fine_food_reviews_with_embeddings_1k.csv\" # for your convenience, we precomputed the embeddings\n",
|
||||
"df = pd.read_csv(datafile_path)\n",
|
||||
"df[\"ada_search\"] = df.ada_search.apply(eval).apply(np.array)\n"
|
||||
"df[\"babbage_search\"] = df.babbage_search.apply(eval).apply(np.array)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -41,7 +39,7 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Good Buy: I liked the beans. They were vacuum sealed, plump and moist. Would recommend them for any use. I personally split and stuck them in some vodka to make vanilla extract. Yum!\n",
|
||||
"Fantastic Instant Refried beans: Fantastic Instant Refried Beans have been a staple for my family now for nearly 20 years. All 7 of us love it and my grown kids are passing on the tradition.\n",
|
||||
"\n",
|
||||
"Jamaican Blue beans: Excellent coffee bean for roasting. Our family just purchased another 5 pounds for more roasting. Plenty of flavor and mild on acidity when roasted to a dark brown bean and befor\n",
|
||||
"\n",
|
||||
@ -57,9 +55,9 @@
|
||||
"def search_reviews(df, product_description, n=3, pprint=True):\n",
|
||||
" embedding = get_embedding(\n",
|
||||
" product_description,\n",
|
||||
" engine=\"text-embedding-ada-002\"\n",
|
||||
" engine=\"text-search-babbage-query-001\"\n",
|
||||
" )\n",
|
||||
" df[\"similarities\"] = df.ada_search.apply(lambda x: cosine_similarity(x, embedding))\n",
|
||||
" df[\"similarities\"] = df.babbage_search.apply(lambda x: cosine_similarity(x, embedding))\n",
|
||||
"\n",
|
||||
" res = (\n",
|
||||
" df.sort_values(\"similarities\", ascending=False)\n",
|
||||
@ -86,17 +84,17 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Tasty and Quick Pasta: Barilla Whole Grain Fusilli with Vegetable Marinara is tasty and has an excellent chunky vegetable marinara. I just wish there was more of it. If you aren't starving or on a \n",
|
||||
"\n",
|
||||
"sooo good: tastes so good. Worth the money. My boyfriend hates wheat pasta and LOVES this. cooks fast tastes great.I love this brand and started buying more of their pastas. Bulk is best.\n",
|
||||
"\n",
|
||||
"Handy: Love the idea of ready in a minute pasta and for that alone this product gets praise. The pasta is whole grain so that's a big plus and it actually comes out al dente. The vegetable marinara\n",
|
||||
"Tasty and Quick Pasta: Barilla Whole Grain Fusilli with Vegetable Marinara is tasty and has an excellent chunky vegetable marinara. I just wish there was more of it. If you aren't starving or on a \n",
|
||||
"\n",
|
||||
"Rustichella ROCKS!: Anything this company makes is worthwhile eating! My favorite is their Trenne.<br />Their whole wheat pasta is the best I have ever had.\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_reviews(df, \"whole wheat pasta\", n=3)"
|
||||
"res = search_reviews(df, \"whole wheat pasta\", n=3)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -121,7 +119,7 @@
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_reviews(df, \"bad delivery\", n=1)"
|
||||
"res = search_reviews(df, \"bad delivery\", n=1)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -146,7 +144,7 @@
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_reviews(df, \"spoilt\", n=1)"
|
||||
"res = search_reviews(df, \"spoilt\", n=1)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -160,21 +158,21 @@
|
||||
"text": [
|
||||
"Good food: The only dry food my queen cat will eat. Helps prevent hair balls. Good packaging. Arrives promptly. Recommended by a friend who sells pet food.\n",
|
||||
"\n",
|
||||
"The cats like it: My 7 cats like this food but it is a little yucky for the human. Pieces of mackerel swimming in a dark broth. It is billed as a \"complete\" food and contains carrots, peas and pasta.\n",
|
||||
"Good product: I like that this is a better product for my pets but really for the price of it I couldn't afford to buy this all the time. My cat isn't very picky usually and she ate this, we usually \n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = search_reviews(df, \"pet food\", n=2)"
|
||||
"res = search_reviews(df, \"pet food\", n=2)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "openai-cookbook",
|
||||
"display_name": "Python 3.9.9 ('openai')",
|
||||
"language": "python",
|
||||
"name": "openai-cookbook"
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
@ -186,12 +184,12 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.6"
|
||||
"version": "3.9.9"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
|
||||
"hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@ -1,452 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Unit test writing using a multi-step prompt\n",
|
||||
"\n",
|
||||
"Complex tasks, such as writing unit tests, can benefit from multi-step prompts. In contrast to a single prompt, a multi-step prompt generates text from GPT-3 and then feeds that text back into subsequent prompts. This can help in cases where you want GPT-3 to explain its reasoning before answering, or brainstorm a plan before executing it.\n",
|
||||
"\n",
|
||||
"In this notebook, we use a 3-step prompt to write unit tests in Python using the following steps:\n",
|
||||
"\n",
|
||||
"1. Given a Python function, we first prompt GPT-3 to explain what the function is doing.\n",
|
||||
"2. Second, we prompt GPT-3 to plan a set of unit tests for the function.\n",
|
||||
" - If the plan is too short, we ask GPT-3 to elaborate with more ideas for unit tests.\n",
|
||||
"3. Finally, we prompt GPT-3 to write the unit tests.\n",
|
||||
"\n",
|
||||
"The code example illustrates a few optional embellishments on the chained, multi-step prompt:\n",
|
||||
"\n",
|
||||
"- Conditional branching (e.g., only asking for elaboration if the first plan is too short)\n",
|
||||
"- Different models for different steps (e.g., `text-davinci-002` for the text planning steps and `code-davinci-002` for the code writing step)\n",
|
||||
"- A check that re-runs the function if the output is unsatisfactory (e.g., if the output code cannot be parsed by Python's `ast` module)\n",
|
||||
"- Streaming output so that you can start reading the output before it's fully generated (useful for long, multi-step outputs)\n",
|
||||
"\n",
|
||||
"The full 3-step prompt looks like this (using as an example `pytest` for the unit test framework and `is_palindrome` as the function):\n",
|
||||
"\n",
|
||||
" # How to write great unit tests with pytest\n",
|
||||
"\n",
|
||||
" In this advanced tutorial for experts, we'll use Python 3.9 and `pytest` to write a suite of unit tests to verify the behavior of the following function.\n",
|
||||
" ```python\n",
|
||||
" def is_palindrome(s):\n",
|
||||
" return s == s[::-1]\n",
|
||||
" ```\n",
|
||||
"\n",
|
||||
" Before writing any unit tests, let's review what each element of the function is doing exactly and what the author's intentions may have been.\n",
|
||||
" - First,{GENERATED IN STEP 1}\n",
|
||||
" \n",
|
||||
" A good unit test suite should aim to:\n",
|
||||
" - Test the function's behavior for a wide range of possible inputs\n",
|
||||
" - Test edge cases that the author may not have foreseen\n",
|
||||
" - Take advantage of the features of `pytest` to make the tests easy to write and maintain\n",
|
||||
" - Be easy to read and understand, with clean code and descriptive names\n",
|
||||
" - Be deterministic, so that the tests always pass or fail in the same way\n",
|
||||
"\n",
|
||||
" `pytest` has many convenient features that make it easy to write and maintain unit tests. We'll use them to write unit tests for the function above.\n",
|
||||
"\n",
|
||||
" For this particular function, we'll want our unit tests to handle the following diverse scenarios (and under each scenario, we include a few examples as sub-bullets):\n",
|
||||
" -{GENERATED IN STEP 2}\n",
|
||||
"\n",
|
||||
" [OPTIONALLY APPENDED]In addition to the scenarios above, we'll also want to make sure we don't forget to test rare or unexpected edge cases (and under each edge case, we include a few examples as sub-bullets):\n",
|
||||
" -{GENERATED IN STEP 2B}\n",
|
||||
"\n",
|
||||
" Before going into the individual tests, let's first look at the complete suite of unit tests as a cohesive whole. We've added helpful comments to explain what each line does.\n",
|
||||
" ```python\n",
|
||||
" import pytest # used for our unit tests\n",
|
||||
"\n",
|
||||
" def is_palindrome(s):\n",
|
||||
" return s == s[::-1]\n",
|
||||
"\n",
|
||||
" #Below, each test case is represented by a tuple passed to the @pytest.mark.parametrize decorator\n",
|
||||
" {GENERATED IN STEP 3}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# imports needed to run the code in this notebook\n",
|
||||
"import ast # used for detecting whether generated Python code is valid\n",
|
||||
"import openai # used for calling the OpenAI API\n",
|
||||
"\n",
|
||||
"# example of a function that uses a multi-step prompt to write unit tests\n",
|
||||
"def unit_test_from_function(\n",
|
||||
" function_to_test: str, # Python function to test, as a string\n",
|
||||
" unit_test_package: str = \"pytest\", # unit testing package; use the name as it appears in the import statement\n",
|
||||
" approx_min_cases_to_cover: int = 7, # minimum number of test case categories to cover (approximate)\n",
|
||||
" print_text: bool = False, # optionally prints text; helpful for understanding the function & debugging\n",
|
||||
" text_model: str = \"text-davinci-002\", # model used to generate text plans in steps 1, 2, and 2b\n",
|
||||
" code_model: str = \"code-davinci-002\", # if you don't have access to code models, you can use text models here instead\n",
|
||||
" max_tokens: int = 1000, # can set this high, as generations should be stopped earlier by stop sequences\n",
|
||||
" temperature: float = 0.4, # temperature = 0 can sometimes get stuck in repetitive loops, so we use 0.4\n",
|
||||
" reruns_if_fail: int = 1, # if the output code cannot be parsed, this will re-run the function up to N times\n",
|
||||
") -> str:\n",
|
||||
" \"\"\"Outputs a unit test for a given Python function, using a 3-step GPT-3 prompt.\"\"\"\n",
|
||||
"\n",
|
||||
" # Step 1: Generate an explanation of the function\n",
|
||||
"\n",
|
||||
" # create a markdown-formatted prompt that asks GPT-3 to complete an explanation of the function, formatted as a bullet list\n",
|
||||
" prompt_to_explain_the_function = f\"\"\"# How to write great unit tests with {unit_test_package}\n",
|
||||
"\n",
|
||||
"In this advanced tutorial for experts, we'll use Python 3.9 and `{unit_test_package}` to write a suite of unit tests to verify the behavior of the following function.\n",
|
||||
"```python\n",
|
||||
"{function_to_test}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Before writing any unit tests, let's review what each element of the function is doing exactly and what the author's intentions may have been.\n",
|
||||
"- First,\"\"\"\n",
|
||||
" if print_text:\n",
|
||||
" text_color_prefix = \"\\033[30m\" # black; if you read against a dark background \\033[97m is white\n",
|
||||
" print(text_color_prefix + prompt_to_explain_the_function, end=\"\") # end='' prevents a newline from being printed\n",
|
||||
"\n",
|
||||
" # send the prompt to the API, using \\n\\n as a stop sequence to stop at the end of the bullet list\n",
|
||||
" explanation_response = openai.Completion.create(\n",
|
||||
" model=text_model,\n",
|
||||
" prompt=prompt_to_explain_the_function,\n",
|
||||
" stop=[\"\\n\\n\", \"\\n\\t\\n\", \"\\n \\n\"],\n",
|
||||
" max_tokens=max_tokens,\n",
|
||||
" temperature=temperature,\n",
|
||||
" stream=True,\n",
|
||||
" )\n",
|
||||
" explanation_completion = \"\"\n",
|
||||
" if print_text:\n",
|
||||
" completion_color_prefix = \"\\033[92m\" # green\n",
|
||||
" print(completion_color_prefix, end=\"\")\n",
|
||||
" for event in explanation_response:\n",
|
||||
" event_text = event[\"choices\"][0][\"text\"]\n",
|
||||
" explanation_completion += event_text\n",
|
||||
" if print_text:\n",
|
||||
" print(event_text, end=\"\")\n",
|
||||
"\n",
|
||||
" # Step 2: Generate a plan to write a unit test\n",
|
||||
"\n",
|
||||
" # create a markdown-formatted prompt that asks GPT-3 to complete a plan for writing unit tests, formatted as a bullet list\n",
|
||||
" prompt_to_explain_a_plan = f\"\"\"\n",
|
||||
" \n",
|
||||
"A good unit test suite should aim to:\n",
|
||||
"- Test the function's behavior for a wide range of possible inputs\n",
|
||||
"- Test edge cases that the author may not have foreseen\n",
|
||||
"- Take advantage of the features of `{unit_test_package}` to make the tests easy to write and maintain\n",
|
||||
"- Be easy to read and understand, with clean code and descriptive names\n",
|
||||
"- Be deterministic, so that the tests always pass or fail in the same way\n",
|
||||
"\n",
|
||||
"`{unit_test_package}` has many convenient features that make it easy to write and maintain unit tests. We'll use them to write unit tests for the function above.\n",
|
||||
"\n",
|
||||
"For this particular function, we'll want our unit tests to handle the following diverse scenarios (and under each scenario, we include a few examples as sub-bullets):\n",
|
||||
"-\"\"\"\n",
|
||||
" if print_text:\n",
|
||||
" print(text_color_prefix + prompt_to_explain_a_plan, end=\"\")\n",
|
||||
"\n",
|
||||
" # append this planning prompt to the results from step 1\n",
|
||||
" prior_text = prompt_to_explain_the_function + explanation_completion\n",
|
||||
" full_plan_prompt = prior_text + prompt_to_explain_a_plan\n",
|
||||
"\n",
|
||||
" # send the prompt to the API, using \\n\\n as a stop sequence to stop at the end of the bullet list\n",
|
||||
" plan_response = openai.Completion.create(\n",
|
||||
" model=text_model,\n",
|
||||
" prompt=full_plan_prompt,\n",
|
||||
" stop=[\"\\n\\n\", \"\\n\\t\\n\", \"\\n \\n\"],\n",
|
||||
" max_tokens=max_tokens,\n",
|
||||
" temperature=temperature,\n",
|
||||
" stream=True,\n",
|
||||
" )\n",
|
||||
" plan_completion = \"\"\n",
|
||||
" if print_text:\n",
|
||||
" print(completion_color_prefix, end=\"\")\n",
|
||||
" for event in plan_response:\n",
|
||||
" event_text = event[\"choices\"][0][\"text\"]\n",
|
||||
" plan_completion += event_text\n",
|
||||
" if print_text:\n",
|
||||
" print(event_text, end=\"\")\n",
|
||||
"\n",
|
||||
" # Step 2b: If the plan is short, ask GPT-3 to elaborate further\n",
|
||||
" # this counts top-level bullets (e.g., categories), but not sub-bullets (e.g., test cases)\n",
|
||||
" elaboration_needed = plan_completion.count(\"\\n-\") +1 < approx_min_cases_to_cover # adds 1 because the first bullet is not counted\n",
|
||||
" if elaboration_needed:\n",
|
||||
" prompt_to_elaborate_on_the_plan = f\"\"\"\n",
|
||||
"\n",
|
||||
"In addition to the scenarios above, we'll also want to make sure we don't forget to test rare or unexpected edge cases (and under each edge case, we include a few examples as sub-bullets):\n",
|
||||
"-\"\"\"\n",
|
||||
" if print_text:\n",
|
||||
" print(text_color_prefix + prompt_to_elaborate_on_the_plan, end=\"\")\n",
|
||||
"\n",
|
||||
" # append this elaboration prompt to the results from step 2\n",
|
||||
" prior_text = full_plan_prompt + plan_completion\n",
|
||||
" full_elaboration_prompt = prior_text + prompt_to_elaborate_on_the_plan\n",
|
||||
"\n",
|
||||
" # send the prompt to the API, using \\n\\n as a stop sequence to stop at the end of the bullet list\n",
|
||||
" elaboration_response = openai.Completion.create(\n",
|
||||
" model=text_model,\n",
|
||||
" prompt=full_elaboration_prompt,\n",
|
||||
" stop=[\"\\n\\n\", \"\\n\\t\\n\", \"\\n \\n\"],\n",
|
||||
" max_tokens=max_tokens,\n",
|
||||
" temperature=temperature,\n",
|
||||
" stream=True,\n",
|
||||
" )\n",
|
||||
" elaboration_completion = \"\"\n",
|
||||
" if print_text:\n",
|
||||
" print(completion_color_prefix, end=\"\")\n",
|
||||
" for event in elaboration_response:\n",
|
||||
" event_text = event[\"choices\"][0][\"text\"]\n",
|
||||
" elaboration_completion += event_text\n",
|
||||
" if print_text:\n",
|
||||
" print(event_text, end=\"\")\n",
|
||||
"\n",
|
||||
" # Step 3: Generate the unit test\n",
|
||||
"\n",
|
||||
" # create a markdown-formatted prompt that asks GPT-3 to complete a unit test\n",
|
||||
" starter_comment = \"\"\n",
|
||||
" if unit_test_package == \"pytest\":\n",
|
||||
" starter_comment = \"Below, each test case is represented by a tuple passed to the @pytest.mark.parametrize decorator\"\n",
|
||||
" prompt_to_generate_the_unit_test = f\"\"\"\n",
|
||||
"\n",
|
||||
"Before going into the individual tests, let's first look at the complete suite of unit tests as a cohesive whole. We've added helpful comments to explain what each line does.\n",
|
||||
"```python\n",
|
||||
"import {unit_test_package} # used for our unit tests\n",
|
||||
"\n",
|
||||
"{function_to_test}\n",
|
||||
"\n",
|
||||
"#{starter_comment}\"\"\"\n",
|
||||
" if print_text:\n",
|
||||
" print(text_color_prefix + prompt_to_generate_the_unit_test, end=\"\")\n",
|
||||
"\n",
|
||||
" # append this unit test prompt to the results from step 3\n",
|
||||
" if elaboration_needed:\n",
|
||||
" prior_text = full_elaboration_prompt + elaboration_completion\n",
|
||||
" else:\n",
|
||||
" prior_text = full_plan_prompt + plan_completion\n",
|
||||
" full_unit_test_prompt = prior_text + prompt_to_generate_the_unit_test\n",
|
||||
"\n",
|
||||
" # send the prompt to the API, using ``` as a stop sequence to stop at the end of the code block\n",
|
||||
" unit_test_response = openai.Completion.create(\n",
|
||||
" model=code_model,\n",
|
||||
" prompt=full_unit_test_prompt,\n",
|
||||
" stop=\"```\",\n",
|
||||
" max_tokens=max_tokens,\n",
|
||||
" temperature=temperature,\n",
|
||||
" stream=True\n",
|
||||
" )\n",
|
||||
" unit_test_completion = \"\"\n",
|
||||
" if print_text:\n",
|
||||
" print(completion_color_prefix, end=\"\")\n",
|
||||
" for event in unit_test_response:\n",
|
||||
" event_text = event[\"choices\"][0][\"text\"]\n",
|
||||
" unit_test_completion += event_text\n",
|
||||
" if print_text:\n",
|
||||
" print(event_text, end=\"\")\n",
|
||||
"\n",
|
||||
" # check the output for errors\n",
|
||||
" code_start_index = prompt_to_generate_the_unit_test.find(\"```python\\n\") + len(\"```python\\n\")\n",
|
||||
" code_output = prompt_to_generate_the_unit_test[code_start_index:] + unit_test_completion\n",
|
||||
" try:\n",
|
||||
" ast.parse(code_output)\n",
|
||||
" except SyntaxError as e:\n",
|
||||
" print(f\"Syntax error in generated code: {e}\")\n",
|
||||
" if reruns_if_fail > 0:\n",
|
||||
" print(\"Rerunning...\")\n",
|
||||
" return unit_test_from_function(\n",
|
||||
" function_to_test=function_to_test,\n",
|
||||
" unit_test_package=unit_test_package,\n",
|
||||
" approx_min_cases_to_cover=approx_min_cases_to_cover,\n",
|
||||
" print_text=print_text,\n",
|
||||
" text_model=text_model,\n",
|
||||
" code_model=code_model,\n",
|
||||
" max_tokens=max_tokens,\n",
|
||||
" temperature=temperature,\n",
|
||||
" reruns_if_fail=reruns_if_fail-1, # decrement rerun counter when calling again\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # return the unit test as a string\n",
|
||||
" return unit_test_completion\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[30m# How to write great unit tests with pytest\n",
|
||||
"\n",
|
||||
"In this advanced tutorial for experts, we'll use Python 3.9 and `pytest` to write a suite of unit tests to verify the behavior of the following function.\n",
|
||||
"```python\n",
|
||||
"def is_palindrome(s):\n",
|
||||
" return s == s[::-1]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Before writing any unit tests, let's review what each element of the function is doing exactly and what the author's intentions may have been.\n",
|
||||
"- First,\u001b[92m we have a function definition. This is where we give the function a name, `is_palindrome`, and specify the arguments that the function accepts. In this case, the function accepts a single string argument, `s`.\n",
|
||||
"- Next, we have a return statement. This is where we specify the value that the function returns. In this case, the function returns `s == s[::-1]`.\n",
|
||||
"- Finally, we have a function call. This is where we actually call the function with a specific set of arguments. In this case, we're calling the function with the string `\"racecar\"`.\u001b[30m\n",
|
||||
" \n",
|
||||
"A good unit test suite should aim to:\n",
|
||||
"- Test the function's behavior for a wide range of possible inputs\n",
|
||||
"- Test edge cases that the author may not have foreseen\n",
|
||||
"- Take advantage of the features of `pytest` to make the tests easy to write and maintain\n",
|
||||
"- Be easy to read and understand, with clean code and descriptive names\n",
|
||||
"- Be deterministic, so that the tests always pass or fail in the same way\n",
|
||||
"\n",
|
||||
"`pytest` has many convenient features that make it easy to write and maintain unit tests. We'll use them to write unit tests for the function above.\n",
|
||||
"\n",
|
||||
"For this particular function, we'll want our unit tests to handle the following diverse scenarios (and under each scenario, we include a few examples as sub-bullets):\n",
|
||||
"-\u001b[92m The input is a palindrome\n",
|
||||
" - `\"racecar\"`\n",
|
||||
" - `\"madam\"`\n",
|
||||
" - `\"anna\"`\n",
|
||||
"- The input is not a palindrome\n",
|
||||
" - `\"python\"`\n",
|
||||
" - `\"test\"`\n",
|
||||
" - `\"1234\"`\n",
|
||||
"- The input is an empty string\n",
|
||||
" - `\"\"`\n",
|
||||
"- The input is `None`\n",
|
||||
"- The input is not a string\n",
|
||||
" - `1`\n",
|
||||
" - `1.0`\n",
|
||||
" - `True`\n",
|
||||
" - `False`\n",
|
||||
" - `[]`\n",
|
||||
" - `{}`\u001b[30m\n",
|
||||
"\n",
|
||||
"In addition to the scenarios above, we'll also want to make sure we don't forget to test rare or unexpected edge cases (and under each edge case, we include a few examples as sub-bullets):\n",
|
||||
"-\u001b[92m The input is a palindrome with spaces\n",
|
||||
" - `\"race car\"`\n",
|
||||
" - `\" madam \"`\n",
|
||||
" - `\" anna \"`\n",
|
||||
"- The input is not a palindrome with spaces\n",
|
||||
" - `\" python \"`\n",
|
||||
" - `\" test \"`\n",
|
||||
" - `\" 1234 \"`\n",
|
||||
"- The input is a palindrome with punctuation\n",
|
||||
" - `\"racecar!\"`\n",
|
||||
" - `\"Madam, I'm Adam.\"`\n",
|
||||
" - `\"Anna's\"`\n",
|
||||
"- The input is not a palindrome with punctuation\n",
|
||||
" - `\"python!\"`\n",
|
||||
" - `\"test.\"`\n",
|
||||
" - `\"1234!\"`\n",
|
||||
"- The input is a palindrome with mixed case\n",
|
||||
" - `\"Racecar\"`\n",
|
||||
" - `\"Madam\"`\n",
|
||||
" - `\"Anna\"`\n",
|
||||
"- The input is not a palindrome with mixed case\n",
|
||||
" - `\"Python\"`\n",
|
||||
" - `\"Test\"`\n",
|
||||
" - `\"1234\"`\u001b[30m\n",
|
||||
"\n",
|
||||
"Before going into the individual tests, let's first look at the complete suite of unit tests as a cohesive whole. We've added helpful comments to explain what each line does.\n",
|
||||
"```python\n",
|
||||
"import pytest # used for our unit tests\n",
|
||||
"\n",
|
||||
"def is_palindrome(s):\n",
|
||||
" return s == s[::-1]\n",
|
||||
"\n",
|
||||
"#Below, each test case is represented by a tuple passed to the @pytest.mark.parametrize decorator\u001b[92m.\n",
|
||||
"#The first element of the tuple is a name for the test case, and the second element is a list of arguments for the test case.\n",
|
||||
"#The @pytest.mark.parametrize decorator will generate a separate test function for each test case.\n",
|
||||
"#The generated test function will be named test_is_palindrome_<name> where <name> is the name of the test case.\n",
|
||||
"#The generated test function will be given the arguments specified in the list of arguments for the test case.\n",
|
||||
"#The generated test function will be given the fixture specified in the decorator, in this case the function itself.\n",
|
||||
"#The generated test function will call the function with the arguments and assert that the result is equal to the expected value.\n",
|
||||
"@pytest.mark.parametrize(\n",
|
||||
" \"name,args,expected\",\n",
|
||||
" [\n",
|
||||
" # Test the function's behavior for a wide range of possible inputs\n",
|
||||
" (\"palindrome\", [\"racecar\"], True),\n",
|
||||
" (\"palindrome\", [\"madam\"], True),\n",
|
||||
" (\"palindrome\", [\"anna\"], True),\n",
|
||||
" (\"non-palindrome\", [\"python\"], False),\n",
|
||||
" (\"non-palindrome\", [\"test\"], False),\n",
|
||||
" (\"non-palindrome\", [\"1234\"], False),\n",
|
||||
" (\"empty string\", [\"\"], True),\n",
|
||||
" (\"None\", [None], False),\n",
|
||||
" (\"non-string\", [1], False),\n",
|
||||
" (\"non-string\", [1.0], False),\n",
|
||||
" (\"non-string\", [True], False),\n",
|
||||
" (\"non-string\", [False], False),\n",
|
||||
" (\"non-string\", [[]], False),\n",
|
||||
" (\"non-string\", [{}], False),\n",
|
||||
" # Test edge cases that the author may not have foreseen\n",
|
||||
" (\"palindrome with spaces\", [\"race car\"], True),\n",
|
||||
" (\"palindrome with spaces\", [\" madam \"], True),\n",
|
||||
" (\"palindrome with spaces\", [\" anna \"], True),\n",
|
||||
" (\"non-palindrome with spaces\", [\" python \"], False),\n",
|
||||
" (\"non-palindrome with spaces\", [\" test \"], False),\n",
|
||||
" (\"non-palindrome with spaces\", [\" 1234 \"], False),\n",
|
||||
" (\"palindrome with punctuation\", [\"racecar!\"], True),\n",
|
||||
" (\"palindrome with punctuation\", [\"Madam, I'm Adam.\"], True),\n",
|
||||
" (\"palindrome with punctuation\", [\"Anna's\"], True),\n",
|
||||
" (\"non-palindrome with punctuation\", [\"python!\"], False),\n",
|
||||
" (\"non-palindrome with punctuation\", [\"test.\"], False),\n",
|
||||
" (\"non-palindrome with punctuation\", [\"1234!\"], False),\n",
|
||||
" (\"palindrome with mixed case\", [\"Racecar\"], True),\n",
|
||||
" (\"palindrome with mixed case\", [\"Madam\"], True),\n",
|
||||
" (\"palindrome with mixed case\", [\"Anna\"], True),\n",
|
||||
" (\"non-palindrome with mixed case\", [\"Python\"], False),\n",
|
||||
" (\"non-palindrome with mixed case\", [\"Test\"], False),\n",
|
||||
" (\"non-palindrome with mixed case\", [\"1234\"], False),\n",
|
||||
" ],\n",
|
||||
")\n",
|
||||
"def test_is_palindrome(is_palindrome, args, expected):\n",
|
||||
" assert is_palindrome(*args) == expected\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'.\\n#The first element of the tuple is a name for the test case, and the second element is a list of arguments for the test case.\\n#The @pytest.mark.parametrize decorator will generate a separate test function for each test case.\\n#The generated test function will be named test_is_palindrome_<name> where <name> is the name of the test case.\\n#The generated test function will be given the arguments specified in the list of arguments for the test case.\\n#The generated test function will be given the fixture specified in the decorator, in this case the function itself.\\n#The generated test function will call the function with the arguments and assert that the result is equal to the expected value.\\n@pytest.mark.parametrize(\\n \"name,args,expected\",\\n [\\n # Test the function\\'s behavior for a wide range of possible inputs\\n (\"palindrome\", [\"racecar\"], True),\\n (\"palindrome\", [\"madam\"], True),\\n (\"palindrome\", [\"anna\"], True),\\n (\"non-palindrome\", [\"python\"], False),\\n (\"non-palindrome\", [\"test\"], False),\\n (\"non-palindrome\", [\"1234\"], False),\\n (\"empty string\", [\"\"], True),\\n (\"None\", [None], False),\\n (\"non-string\", [1], False),\\n (\"non-string\", [1.0], False),\\n (\"non-string\", [True], False),\\n (\"non-string\", [False], False),\\n (\"non-string\", [[]], False),\\n (\"non-string\", [{}], False),\\n # Test edge cases that the author may not have foreseen\\n (\"palindrome with spaces\", [\"race car\"], True),\\n (\"palindrome with spaces\", [\" madam \"], True),\\n (\"palindrome with spaces\", [\" anna \"], True),\\n (\"non-palindrome with spaces\", [\" python \"], False),\\n (\"non-palindrome with spaces\", [\" test \"], False),\\n (\"non-palindrome with spaces\", [\" 1234 \"], False),\\n (\"palindrome with punctuation\", [\"racecar!\"], True),\\n (\"palindrome with punctuation\", [\"Madam, I\\'m Adam.\"], True),\\n (\"palindrome with punctuation\", [\"Anna\\'s\"], True),\\n (\"non-palindrome with punctuation\", [\"python!\"], False),\\n (\"non-palindrome with punctuation\", [\"test.\"], False),\\n (\"non-palindrome with punctuation\", [\"1234!\"], False),\\n (\"palindrome with mixed case\", [\"Racecar\"], True),\\n (\"palindrome with mixed case\", [\"Madam\"], True),\\n (\"palindrome with mixed case\", [\"Anna\"], True),\\n (\"non-palindrome with mixed case\", [\"Python\"], False),\\n (\"non-palindrome with mixed case\", [\"Test\"], False),\\n (\"non-palindrome with mixed case\", [\"1234\"], False),\\n ],\\n)\\ndef test_is_palindrome(is_palindrome, args, expected):\\n assert is_palindrome(*args) == expected\\n'"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"example_function = \"\"\"def is_palindrome(s):\n",
|
||||
" return s == s[::-1]\"\"\"\n",
|
||||
"\n",
|
||||
"unit_test_from_function(example_function, print_text=True)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.9 ('openai')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.9"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -75,7 +75,7 @@
|
||||
"\n",
|
||||
"# Now let's create the deployment\n",
|
||||
"print(f'Creating a new deployment with model: {model}')\n",
|
||||
"result = openai.Deployment.create(model=model, scale_settings={\"scale_type\":\"standard\"})\n",
|
||||
"result = openai.Deployment.create(model=model, scale_settings={\"scale_type\":\"manual\", \"capacity\": 1})\n",
|
||||
"deployment_id = result[\"id\"]"
|
||||
]
|
||||
},
|
||||
|
||||
@ -58,7 +58,7 @@
|
||||
"from azure.identity import DefaultAzureCredential\n",
|
||||
"\n",
|
||||
"default_credential = DefaultAzureCredential()\n",
|
||||
"token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n",
|
||||
"token = default_credential.get_token(\"https://cognitiveservices.azure.com\")\n",
|
||||
"\n",
|
||||
"openai.api_type = 'azure_ad'\n",
|
||||
"openai.api_key = token.token\n",
|
||||
@ -346,7 +346,7 @@
|
||||
"\n",
|
||||
"# Now let's create the deployment\n",
|
||||
"print(f'Creating a new deployment with model: {model}')\n",
|
||||
"result = openai.Deployment.create(model=model, scale_settings={\"scale_type\":\"standard\"})\n",
|
||||
"result = openai.Deployment.create(model=model, scale_settings={\"scale_type\":\"manual\", \"capacity\": 1})\n",
|
||||
"deployment_id = result[\"id\"]\n"
|
||||
]
|
||||
},
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -1,362 +0,0 @@
|
||||
Date,Supplier,Description,Transaction value (£)
|
||||
21/04/2016,M & J Ballantyne Ltd,George IV Bridge Work,35098
|
||||
26/04/2016,Private Sale,Literary & Archival Items,30000
|
||||
30/04/2016,City Of Edinburgh Council,Non Domestic Rates ,40800
|
||||
09/05/2016,Computacenter Uk,Kelvin Hall,72835
|
||||
09/05/2016,John Graham Construction Ltd,Causewayside Refurbishment,64361
|
||||
09/05/2016,A McGillivray,Causewayside Refurbishment,53690
|
||||
16/05/2016,John Graham Construction Ltd,Causewayside Refurbishment,365344
|
||||
23/05/2016,Computacenter Uk,Kelvin Hall,26506
|
||||
23/05/2016,ECG Facilities Service,Facilities Management Charge,32777
|
||||
23/05/2016,ECG Facilities Service,Facilities Management Charge,32777
|
||||
30/05/2016,ALDL,ALDL Charges,32317
|
||||
10/06/2016,Wavetek Ltd,Kelvin Hall,87589
|
||||
10/06/2016,John Graham Construction Ltd,Causewayside Refurbishment,381803
|
||||
28/06/2016,ECG Facilities Service,Facilities Management Charge,32832
|
||||
30/06/2016,Glasgow City Council,Kelvin Hall,1700000
|
||||
11/07/2016,Wavetek Ltd,Kelvin Hall,65692
|
||||
11/07/2016,John Graham Construction Ltd,Causewayside Refurbishment,139845
|
||||
15/07/2016,Sotheby'S,Literary & Archival Items,28500
|
||||
18/07/2016,Christies,Literary & Archival Items,33800
|
||||
25/07/2016,A McGillivray,Causewayside Refurbishment,30113
|
||||
31/07/2016,ALDL,ALDL Charges,32317
|
||||
08/08/2016,ECG Facilities Service,Facilities Management Charge,32795
|
||||
15/08/2016,Creative Video Productions Ltd,Kelvin Hall,26866
|
||||
15/08/2016,John Graham Construction Ltd,Causewayside Refurbishment,196807
|
||||
24/08/2016,ECG Facilities Service,Facilities Management Charge,32795
|
||||
05/09/2016,John Graham Construction Ltd,Causewayside Refurbishment,36359
|
||||
12/09/2016,Flexiform,Kelvin Hall,42623
|
||||
12/09/2016,City Of Edinburgh Council,Non Domestic Rates ,144330
|
||||
12/09/2016,City Of Edinburgh Council,Non Domestic Rates ,49827
|
||||
12/09/2016,John Graham Construction Ltd,Causewayside Refurbishment,228689
|
||||
19/09/2016,Jisc Services Ltd Subscription Account,Literary & Archival Items,42629
|
||||
26/09/2016,Senator International,Kelvin Hall,35706
|
||||
26/09/2016,ECG Facilities Service,Facilities Management Charge,32795
|
||||
26/09/2016,John Graham Construction Ltd,Causewayside Refurbishment,28378
|
||||
30/09/2016,A McGillivray,Causewayside Refurbishment,44392
|
||||
10/10/2016,Cengage Learning (Emea )Ltd,Literary & Archival Items,86604
|
||||
10/10/2016,John Graham Construction Ltd,Causewayside Refurbishment,303999
|
||||
24/10/2016,ECG Facilities Service,Facilities Management Charge,32795
|
||||
24/10/2016,ALDL,ALDL Charges,32317
|
||||
31/10/2016,John Graham Construction Ltd,Causewayside Refurbishment,74245
|
||||
07/11/2016,CBRE,Kelvin Hall,83736
|
||||
14/11/2016,University Of Glasgow,Kelvin Hall,188682
|
||||
14/11/2016,John Graham Construction Ltd,Causewayside Refurbishment,362326
|
||||
08/12/2016,Sothebys,Literary & Archival Items,166000
|
||||
08/12/2016,Private Sale,Literary & Archival Items,87500
|
||||
08/12/2016,ECG Facilities Service,Facilities Management Charge,32795
|
||||
12/12/2016,John Graham Construction Ltd,Causewayside Refurbishment,385310
|
||||
30/12/2016,ECG Facilities Service,Facilities Management Charge,32795
|
||||
30/12/2016,John Graham Construction Ltd,Causewayside Refurbishment,253618
|
||||
30/12/2016,John Graham Construction Ltd,Causewayside Refurbishment,45127
|
||||
23/01/2017,ALDL,ALDL Charges,27730
|
||||
07/02/2017,ECG Facilities Service,Facilities Management Charge,32795
|
||||
07/02/2017,John Graham Construction Ltd,Causewayside Refurbishment,52404
|
||||
13/02/2017,John Graham Construction Ltd,Causewayside Refurbishment,272390
|
||||
27/02/2017,Cengage Learning (Emea )Ltd,Literary & Archival Items,43302
|
||||
27/02/2017,ECG Facilities Service,Facilities Management Charge,32795
|
||||
06/03/2017,Private Sale,Literary & Archival Items,72500
|
||||
06/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,31781
|
||||
06/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,198048
|
||||
27/03/2017,ECG Facilities Service,Facilities Management Charge,32795
|
||||
31/03/2017,NLS Foundation,Grant Payment,177500
|
||||
31/03/2017,Private Sale,Literary & Archival Items,3422500
|
||||
31/03/2017,Nicholson Bros(Electrical Contractors) Ltd,Causewayside Refurbishment,33666
|
||||
31/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,222090
|
||||
31/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,63971
|
||||
31/03/2017,XMA Scotland Ltd,IT equipment,33450
|
||||
31/03/2017,XMA Scotland Ltd,IT equipment,84524
|
||||
24/04/2017,Cengage Learning (Emea )Ltd,Literary & Archival Items,43302
|
||||
24/04/2017,Scottish Historic Buildings Trust,Lawnmarket Work,50057
|
||||
24/04/2017,Insight Direct (UK) Ltd,IT equipment,56768
|
||||
30/04/2017,Morris & Spottiswood Ltd,George IV Bridge Work,63716
|
||||
08/05/2017,Anglian Water Business,Water,26832
|
||||
15/05/2017,John Graham Construction Ltd,Causewayside Refurbishment,245381
|
||||
22/05/2017,ECG Facilities Service,Facilities Management Charge,33386
|
||||
22/05/2017,ALDL,Legal Deposit Services,27067
|
||||
29/05/2017,ECG Facilities Service,Facilities Management Charge,33386
|
||||
29/05/2017,John Graham Construction Ltd,Causewayside Refurbishment,74806
|
||||
29/05/2017,Morris & Spottiswood Ltd,George IV Bridge Work,56448
|
||||
31/05/2017,John Graham Construction Ltd,Causewayside Refurbishment,164691
|
||||
26/06/2017,ECG Facilities Service,Facilities Management Charge,33386
|
||||
26/06/2017,British Library,Legal Deposit Services,50056
|
||||
24/07/2017,John Graham Construction Ltd,Causewayside Refurbishment,27926
|
||||
24/07/2017,John Graham Construction Ltd,Causewayside Refurbishment,212690
|
||||
24/07/2017,ALDL,Legal Deposit Services,27067
|
||||
24/07/2017,AM Phillip,Vehicle Purchase,26604
|
||||
16/08/2017,ECG Facilities Service,Facilities Management Charge,33386
|
||||
16/08/2017,John Graham Construction Ltd,Causewayside Refurbishment,59021
|
||||
16/08/2017,John Graham Construction Ltd,Causewayside Refurbishment,136379
|
||||
16/08/2017,Ex Libris,IT equipment,76610
|
||||
23/08/2017,Culture And Sport Glasgow,Kelvin Hall,60503
|
||||
23/08/2017,XMA Scotland Ltd,Kelvin Hall,31830
|
||||
23/08/2017,ECG Facilities Service,Facilities Management Charge,33386
|
||||
31/08/2017,John Graham Construction Ltd,Causewayside Refurbishment,36313
|
||||
31/08/2017,Insight Direct (UK) Ltd,Causewayside Refurbishment,68222
|
||||
31/08/2017,Mark Finn Laboratory,George IV Bridge Work,53884
|
||||
11/09/2017,John Graham Construction Ltd,Causewayside Refurbishment,189483
|
||||
15/09/2017,City Of Edinburgh Council,Non Domestic Rates ,57662
|
||||
15/09/2017,City Of Edinburgh Council,Non Domestic Rates ,142680
|
||||
09/10/2017,Frost And Sullivan Ltd,Literary & Archival Items,28125
|
||||
09/10/2017,JISC Services Ltd ,Literary & Archival Items,43481
|
||||
23/10/2017,John Graham Construction Ltd,Causewayside Refurbishment,151659
|
||||
23/10/2017,City Building LLP,Causewayside Refurbishment,53147
|
||||
30/10/2017,ECG Facilities Service,Facilities Management Charge,35758
|
||||
30/10/2017,ECG Facilities Service,Facilities Management Charge,35758
|
||||
06/11/2017,John Graham Construction Ltd,Causewayside Refurbishment,134208
|
||||
06/11/2017,ALDL,Legal Deposit Services,27067
|
||||
27/11/2017,Maggs Bros Ltd,Literary & Archival Items,26500
|
||||
30/11/2017,Glasgow City Council,Kelvin Hall,42345
|
||||
11/12/2017,ECG Facilities Service,Facilities Management Charge,35758
|
||||
11/12/2017,John Graham Construction Ltd,Causewayside Refurbishment,159275
|
||||
08/01/2018,ECG Facilities Service,Facilities Management Charge,35758
|
||||
15/01/2018,Proquest Information And Learn,Literary & Archival Items,42199
|
||||
15/01/2018,John Graham Construction Ltd,Causewayside Refurbishment,123244
|
||||
29/01/2018,ECG Facilities Service,Facilities Management Charge,35758
|
||||
05/02/2018,John Graham Construction Ltd,Causewayside Refurbishment,102659
|
||||
27/02/2018,ALDL,Legal Deposit Services,27067
|
||||
07/03/2018,John Graham Construction Ltd,Causewayside Refurbishment,89559
|
||||
14/03/2018,Bernard Quaritch Ltd,Literary & Archival Items,372500
|
||||
14/03/2018,ECG Facilities Service,Facilities Management Charge,35758
|
||||
21/03/2018,Site Sealants Ltd,Causewayside Refurbishment,27747
|
||||
30/03/2018,Private Sale,Literary & Archival Items,100000
|
||||
30/03/2018,ECG Facilities Service,Facilities Management Charge,35758
|
||||
30/04/2018,ECG FACILITIES SERVICE,Causewayside IT Work,25634.7
|
||||
30/04/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
14/05/2018,GLASGOW CITY COUNCIL,Kelvin Hall,90946
|
||||
11/06/2018,ALDL,ALDL Charges,27067
|
||||
11/06/2018,JOHN GRAHAM CONSTRUCTION LTD,Causewayisde Refurbishment,127753.31
|
||||
22/06/2018,BONHAMS - LONDON,Literary & Archival Items,25025
|
||||
22/06/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
22/06/2018,EX LIBRIS,IT equipment,39000
|
||||
30/06/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
16/07/2018,EX LIBRIS,IT equipment,80057.83
|
||||
18/07/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
18/07/2018,Sotheby's,Literary & Archival Items,41600
|
||||
31/08/2018,AUTOMATED DOCUMENT SERVICES,IT equipment,84480
|
||||
31/08/2018,XMA SCOTLAND LTD,IT equipment,313000
|
||||
13/09/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
13/09/2018,CITY OF EDINBURGH COUNCIL,Non Domestic Rates,59303.2
|
||||
13/09/2018,CITY OF EDINBURGH COUNCIL,Non Domestic Rates,146740
|
||||
20/09/2018,FROST AND SULLIVAN LTD,Literary & Archival Items,28125
|
||||
20/09/2018,SJS Property Services,George IV Bridge Work,44684.2
|
||||
20/09/2018,CENGAGE LEARNING (EMEA )LTD,Literary & Archival Items,64791
|
||||
30/09/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
30/09/2018,SJS Property Services,George IV Bridge Work,51635.35
|
||||
24/10/2018,XMA SCOTLAND LTD,IT equipment,35313.48
|
||||
24/10/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
21/11/2018,EX LIBRIS,IT equipment,39000
|
||||
21/11/2018,EX LIBRIS,IT equipment,53327.09
|
||||
26/11/2018,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
26/11/2018,SJS Property Services,George IV Bridge Work,66818.25
|
||||
11/12/2018,CALEDONIAN LIFT SERVICES LTD,Causewayside Work,47944.8
|
||||
31/12/2018,SOFTCAT,IT equipment,37064.3
|
||||
14/01/2019,m-hance,IT Work,33164.4
|
||||
14/01/2019,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
24/01/2019,ARTHUR MCKAY BUILDING SERVICES,Causewayside Work,100235.17
|
||||
31/01/2019,ECG FACILITIES SERVICE,Causewayside Work,32517.45
|
||||
31/01/2019,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
31/01/2019,CENGAGE LEARNING (EMEA )LTD,Literary & Archival Items,66443
|
||||
14/02/2019,Private Sale,Literary & Archival Items,50000
|
||||
27/02/2019,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
31/03/2019,ECG FACILITIES SERVICE,Facilities Management Charge,35757.91
|
||||
31/03/2019,ECG FACILITIES SERVICE,George IV Bridge Work,37320.15
|
||||
31/03/2019,HP INC UK LTD,IT equipment,40746
|
||||
31/03/2019,INSIGHT DIRECT (UK) LTD,IT equipment,56223.35
|
||||
23/04/2019,EX LIBRIS,"IT equipment
|
||||
",129584.58
|
||||
30/04/2019,ECG FACILITIES SERVICE,Facilities Management Charge,36907.14
|
||||
30/04/2019,COMPUTACENTER UK,"IT equipment
|
||||
",139571.14
|
||||
13/05/2019,GLASGOW LIFE,Kelvin Hall Service Charge,120335
|
||||
04/06/2019,ECG FACILITIES SERVICE,Facilities Management Charge,36907.14
|
||||
24/06/2019,Private Sale,Literary & Archival Items,34400
|
||||
25/06/2019,ECG FACILITIES SERVICE,Facilities Management Charge,36907.14
|
||||
31/07/2019,ECG FACILITIES SERVICE,Facilities Management Charge,36907.14
|
||||
26/08/2019,MICROBOX GmbH,Digital equipment,65881.58
|
||||
27/08/2019,ECG FACILITIES SERVICE,Facilities Management Charge,36907.14
|
||||
27/08/2019,FROST AND SULLIVAN LTD,Literary & Archival Items,28687.5
|
||||
18/09/2019,CITY OF EDINBURGH COUNCIL,Annual Property Rates 2019/20 for three buildings,221467.2
|
||||
25/09/2019,LOTHIAN HEATING SERVICES LTD,Payment 1 - GB Boiler replacement ,57114.18
|
||||
25/09/2019,ECG FACILITIES SERVICE,Facilities Management Charge,34021.61
|
||||
25/09/2019,EDF Energy,Electricity,33122.06
|
||||
18/09/2019,INSTITUTE OF CONSERVATION,Bursary Recruitment and Professional Services costs for intern,26805.2
|
||||
10/10/2019,ECG FACILITIES SERVICE,"CB Bolier Replacement (1),USP Batteries,Gutter Works & Cleaning of pigeon fouling",112794
|
||||
23/10/2019,ECG FACILITIES SERVICE,"CB Bolier Replacement (2),Facilities Management Charge October 19, intumescent strips & unblocking toilets",103462.39
|
||||
23/10/2019,Private Sale,Purchase of Manuscripts,45000
|
||||
04/10/2019,ECG FACILITIES SERVICE,Facilities Management Charge September 19,44288.57
|
||||
10/10/2019,GLASGOW LIFE,Service Charges Kelvin Hall,39100.16
|
||||
15/10/2019,EDF ENERGY,Electricity,26805.74
|
||||
04/10/2019,JISC SERVICES LTD SUBSCRIPTION ACCOUNT,Annual Subscription,25731
|
||||
23/10/2019,ALDL,Oct19-Dec19 charge from Agency for Legal Deposit Libraries,25155.6
|
||||
27/11/2019,ECG FACILITIES SERVICE,"Paymnet for 31 invoices including Facilities Managemenr Charge Nov 19, Lift Repairs, replacement refrigerant gas detection system & data cabling and install of WIFI devices",104526.09
|
||||
05/11/2019,LOTHIAN HEATING SERVICES LTD,GB Bolier Replacement - application 2,45728.9
|
||||
27/11/2019,GLASGOW LIFE,Service Charges Kelvin Hall 01/07/19-30/09/19,41541.47
|
||||
19/11/2019,EDF ENERGY,Electricity Oct 2019 3 buildings,26660.9
|
||||
10/12/2019,PRIVATE SALE,Collection of papers of an individual,125000
|
||||
06/12/2019,PROQUEST,Purchase of 9 subscriptions 01/11/19-31/10/20,61638
|
||||
18/12/2019,ECG,"Payment of 19 separate invoice including for service of chiller, re-route return pipes, data cabling and install of WifI devices, sprinkler work",44556.15
|
||||
22/01/2020,ECG,"Payment of 28 separate invoices including for supply and fit aluminium screen, upgrade boilerhouse electrical panels,CCTV components, pump casting & lift repairs",89297.94
|
||||
09/01/2020,ECG,Payment of 18 separate invoices including for December facilities services and boiler replacement CB,78585.73
|
||||
14/01/2020,LM Information Delivery UK LTD,Payment of 18 separate invoice for Online/Print subscriptions Jan 20-Dec 20,27822.54
|
||||
14/01/2020,EDF,Electricity,25172.34
|
||||
14/01/2020,ALDL,Jan20-Mar 20 charge from Agency for Legal Deposit Libraries,25155.6
|
||||
06/02/2020,XMA Scotland,Scality Ring Maintenance,68464.62
|
||||
06/02/2020,Trustmarque,Miscrosoft Software Licenses,38069.66
|
||||
11/02/2020,Studio MB,Concept Design Semi-Permanent Exhibtion,27000
|
||||
11/02/2020,EDF,Electricity,25484.03
|
||||
06/03/2020,British Library,Governance and Management Costs,27766.6
|
||||
10/03/2020,Proquest,Subscriptions,50309.81
|
||||
10/03/2020,ECG,Two months maintance contracts,80041.02
|
||||
17/03/2020,BSI,Subscription,30951.6
|
||||
17/03/2020,Glasgow Life,Kelvin Hall Service Charges,55857.04
|
||||
17/03/2020,Private Collection,Collection of literary papers,60000
|
||||
20/03/2020,EDF,Electricity,25829.65
|
||||
20/03/2020,ECG,This payment covers 16 invoices including upgrade to boiler control panel & remedial works following 5 year test,32025.98
|
||||
06/04/2020,Gardiner and Theobald,GB Feasibility Study,49508
|
||||
06/04/2020,ECG,This payment covers 8 invocies including monthly facilities management fees & site inspection fees,51822.68
|
||||
23/04/2020,OCLC UK,Cataloging and Metadata subscription,26251.2
|
||||
23/04/2020,John Graham,Stonework Retention Payment,25104.56
|
||||
23/04/2020,EDF,Electricity,25025.89
|
||||
23/04/2020,Studio MB,Exhibition design,63000
|
||||
23/04/2020,ECG,"This payment covers 5 invocies including monthly facilities management fees, software and hardware maintenance & Lighting Upgrades",65200.11
|
||||
14/05/2020,GARDINER AND THEOBALD LLP,GB Feasibility Study,26291.48
|
||||
14/05/2020,HP INC UK LTD,IT equipment purchase,30640.32
|
||||
14/05/2020,XMA SCOTLAND LTD,Purchase of IT equipment and renewal of maintenance agreement. This payment covers 2 invoices,139167.6
|
||||
14/05/2020,CENGAGE LEARNING EMEA LTD,Annual hosting fee,28800
|
||||
21/05/2020,ECG FACILITIES SERVICE,CB Boiler replacement plus monthly maintenance fee. This payment covers 2 invoices,47899.83
|
||||
29/05/2020,EDF ENERGY,Electricity for April in Causewayside and George IV Bridge buildings. This payment covers 2 invoices.,30175.09
|
||||
29/05/2020,SOFTCAT,Software Licence,42866.5
|
||||
09/06/2020,Ex Libris,Annual subsriptions. This payment covers 2 invoices.,189036.11
|
||||
09/06/2020,Glasgow Life,Service Charges,49509.2
|
||||
09/06/2020,XMA Scotland Ltd,IT equipment,25371.84
|
||||
18/06/2020,JISC SERVICES LTD SUBSCRIPTION ACCOUNT,Annual subscription,25896
|
||||
25/06/2020,ECG FACILITIES SERVICE,Facility Management fees,49000
|
||||
25/06/2020,GARDINER AND THEOBALD LLP,GB Feasibility Study,26291.48
|
||||
25/06/2020,THE LEARNING POOL,E-Learning Resources,25344
|
||||
07/07/2020,Agency for the Legal Deposit Libraries,Agency services,26007.95
|
||||
07/07/2020,Lyon and Turnball,Various collection items,54094
|
||||
09/07/2020,XMA Scotland Ltd,Computer equipment,33327
|
||||
14/07/2020,EDF Energy,Utilities,25768.85
|
||||
23/07/2020,Computer Centre UK Ltd,Computer equipment,27750.79
|
||||
23/07/2020,ECG Facility Services,Facility Management fees,49000
|
||||
23/07/2020,GARDINER AND THEOBALD LLP,GB Feasibility Study,26291.48
|
||||
13/08/2020,EDF Energy,Utilities. This transaction is made up of 3 invoices.,26688.27
|
||||
13/08/2020,Frost & Sullivan Ltd,Annual subscription,34425
|
||||
27/08/2020,Agency for Legal Deposit Libaries,Agency services,26007.95
|
||||
27/08/2020,ECG Facilities Services,Facility Management fees,49000
|
||||
27/08/2020,Gardiner and Theobald LLP,GB Feasibility Study,26291.48
|
||||
17/09/2020,EDF Energy,This payment covers 3 invoices for utility services,34283.03
|
||||
17/09/2020,JISC Services Ltd,Subscription,26179.72
|
||||
17/09/2020,XMA Scotland Ltd,IT equipment,26533.92
|
||||
24/09/2020,ECG Facilities Services,Facility Management fees,55450.58
|
||||
24/09/2020,Glasgow Life,Service charges,25211.17
|
||||
08/10/2020,EDF Energy,This payment covers 5 invoices for utility services,27625.53
|
||||
08/10/2020,ALDL,Agency services,26007.95
|
||||
08/10/2020,Institute of Conservation,This payment covers 2 invoices for student bursary costs,31654
|
||||
08/10/2020,Studio MB,Exhibition build works,36000
|
||||
22/10/2020,ECG Facilities,This payment covers 11 invoices for facility Management fees,55672.9
|
||||
22/10/2020,Glasgow City Council,Capital works,34802.4
|
||||
19/11/2020,DTEK DIGITAL SOLUTIONS LTD,Computer equipment,39348
|
||||
19/11/2020,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility Management fees,31888.51
|
||||
19/11/2020,GLASGOW LIFE,Builidng service charges,47690.16
|
||||
26/11/2020,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility Management fees,55299.92
|
||||
26/11/2020,LEE BOYD LIMITED,This payment covers 7 invoices for project management fees,26440.98
|
||||
03/12/2020,PROQUEST INFORMATION AND LEARN,This payment covers multiple invoices for collection items,50232.54
|
||||
10/12/2020,STUDIO MB,This payment covers 2 invoices for exhibition services and equipment,55902
|
||||
17/12/2020,ECG FACILITIES SERVICE,Facility Management Fees,49000
|
||||
17/12/2020,LEE BOYD LIMITED,This payment covers multiple invoices for project management fees,28922.8
|
||||
07/01/2021,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility management fees,39150.26
|
||||
14/01/2021,EDF ENERGY,This payment covers multiple invoices for electricity,28711.17
|
||||
14/01/2021,ALDL,Legal deposit services,26007.95
|
||||
14/01/2021,EXCHANGE COMMUNICATIONS INSTALLATIONS LTD,Telecom services,31878
|
||||
21/01/2021,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility management fees,28797.1
|
||||
28/01/2021,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility management fees,54875.74
|
||||
04/02/2021,PROQUEST INFORMATION AND LEARN,One invoice for collection items,40000
|
||||
18/02/2021,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility management fees,54931.68
|
||||
25/02/2021,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility management fees,51283.39
|
||||
25/02/2021,HP INC UK LTD,IT Equipment,37868.04
|
||||
10/03/2021,BSI,BSOL Modular Subscription,30510
|
||||
16/03/2021,PHOENIX SOFTWARE LTD,IT Hardware plus 5 year licence,74432.04
|
||||
16/03/2021,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility management fees,134758.64
|
||||
23/03/2021,ECG FACILITIES SERVICE,Maintenance Contract - March,49000
|
||||
23/03/2021,ICAM ARCHIVE SYSTEMS,Camera System - phase 1,39120
|
||||
25/03/2021,ECG FACILITIES SERVICE,This payment covers multiple invoices for facility management fees,108450.85
|
||||
31/03/2021,GLASGOW LIFE,Oct 20 to Dec 20 service charge - Kelvin Hall,54840.53
|
||||
31/03/2021,ECG FACILITIES SERVICE,Replacement Humidifer units,76751
|
||||
31/03/2021,ECG FACILITIES SERVICE,Cooling and Humidifer system upgrade,26943.84
|
||||
31/03/2021,ECG FACILITIES SERVICE,Installation of CCTV,29404.62
|
||||
29/04/2021,ECG FACILITIES SERVICE,This payment covers April 21 Maintenance Contract and the installation of battery rack and batteries plus smaller maintenance invoices,71604.07
|
||||
29/04/2021,GLASGOW LIFE,Jan 21 to Mar 21 service charge - Kelvin Hall,46657.33
|
||||
20/05/2021,ECG FACILITIES SERVICE,Routine inspection and maintenance of all NLS properties,52584.2
|
||||
27/05/2021,XMA SCOTLAND LTD,2 invoices one for the replacement of obsolete hardware and the other for a new laptop,28587.59
|
||||
13/05/2021,ALDL,"Claiming, receipting and onward distribution of legal deposit on behalf of NLS",26376.68
|
||||
27/05/2021,LYON AND TURNBULL,Purchase of a manuscript,26000
|
||||
27/05/2021,ARNOLD CLARK,Purchase of an electric van,25949.5
|
||||
28/06/2021,XMA Scotland Ltd,Purchase of IT hardware for cloud and maintenance of hardware,72061.92
|
||||
08/07/2021,EX LIBRIS,Subscription April to Oct 21 cloud based library services,95045.31
|
||||
08/07/2021,ECG FACILITIES SERVICE,Maintenance contract - June 21 period,52459.25
|
||||
08/07/2021,XMA SCOTLAND LTD,IT hardware equipment,37620.86
|
||||
22/07/2021,ALDL,Quarterly invoice legal deposit materials - July to Sept 21,26400.68
|
||||
12/08/2021,ECG FACILITIES SERVICE,Maintenance contract - July 21 period,52459.25
|
||||
27/08/2021,ECG FACILITIES SERVICE,Maintenance contract - August 21 period,52459.25
|
||||
27/08/2021,ECG FACILITIES SERVICE,Water penetration works - part 2,28350
|
||||
27/08/2021,ECG FACILITIES SERVICE,Water penetration works - part 3,28350
|
||||
22/09/2021,GLASGOW LIFE,Kelvin Hall Service Charge - April to June 21,35420.45
|
||||
29/09/2021,ECG FACILITIES SERVICE,Maintenance contract - all properties,52459.25
|
||||
29/09/2021,FROST AND SULLIVAN LTD,Annual Subscription - Sept 21 to Oct 22,35147.09
|
||||
21/10/2021,ECG FACILITIES SERVICE,Maintenance contract - October,52459.25
|
||||
31/10/2021,SOFTCAT,It purchases for server,42282.72
|
||||
14/10/2021,ALDL,"Claiming, receipting and onward distribution for quarter Oct to Dec 21",26400.68
|
||||
04/11/2021,Web of Science JISC SHEDL subs ,Subscription 2021 to 2021 SHEDL,28361.78
|
||||
11/11/2021,M and J Kelman Ltd,Literary and personal papers of James Kelman,40000
|
||||
11/11/2021,John Graham Constrution Ltd,External fabric repairs - Causeway Side building,75262.75
|
||||
11/11/2021,Robert Harland,Correspondance and Literary papers - Thomas Carlyle,94000
|
||||
11/11/2021,Jisc Services Ltd,IT Subscription and router service charge,25896
|
||||
25/11/2021,ECG Facilities,Maintenance Contract - November,52459.25
|
||||
25/11/2021,Ex Libris,IT Subscription ,81729.02
|
||||
31/12/2021,ECG FACILITIES SERVICE,Electrical and mechanical works,28071.17
|
||||
16/12/2021,JAMES BRECK LTD,Re-slating of roof LB,28572.28
|
||||
23/12/2021,CENGAGE LEARNING EMEA LTD,Subscription - Historical Archive,32460
|
||||
31/12/2021,GLASGOW LIFE,Quarterly service charge KH,45541.34
|
||||
31/12/2021,ECG FACILITIES SERVICE,Maintenance Contract - December,52459.25
|
||||
16/12/2021,ECG FACILITIES SERVICE,"Electrical, mechanical and building works",82227.96
|
||||
27/01/2022,ECG FACILITIES SERVICE,January maintenance contract,52459.25
|
||||
31/01/2022,ALDL,1st January to 31st March 22 - receipting and onward distribution of UK legal deposit materials on behalf of National Library of Scotland,26388.68
|
||||
03/02/2022,ECG FACILITIES SERVICE,"Monthly maintenance contract, drainage jetting and cctv remedials, patio roofing wash",62411.69
|
||||
10/02/2022,JAMES BRECK LTD,Roof uplifting and re-slating,31890.41
|
||||
10/02/2022,LEE BOYD LIMITED,Various invoices smoke extract system and rateable value review,30552
|
||||
17/02/2022,LEE BOYD LIMITED,"Various invoices for CB smoke extract system, project work - FM maintenance framework, sprinkler system",57766.9
|
||||
24/02/2022,ECG FACILITIES SERVICE,"Carry out tanking works, supply and fit mini drive unit, balustrade repairs",27723.16
|
||||
24/02/2022,ADAM MATTHEW DIGITAL LTD,Resource - slavery abolution and social justice,37080
|
||||
10/03/2022,ECG FACILITIES SERVICE,Maintenance contract - March,52459.25
|
||||
10/03/2022,XMA SCOTLAND LTD,It equipment,61885.56
|
||||
17/03/2022,EDF ENERGY,Electricity bill for various sites,57220.55
|
||||
17/03/2022,ECG FACILITIES SERVICE,Maintenance contract - Feb plus various smaller invoices for maintenance jobs,71653.47
|
||||
17/03/2022,XMA010,IT equipment,77208.77
|
||||
17/03/2022,OXFORD UNIVERSITY PRESS,Annual subscription,28576.89
|
||||
24/03/2022,ECG FACILITIES SERVICE,Various small maintenance jobs around library sites,34055.73
|
||||
24/03/2022,GLASGOW LIFE,Kelvin Hall quarterly service charge,41637.96
|
||||
24/03/2022,LEE BOYD LIMITED,Sprinkler system project and lift refurb George IV,55234
|
||||
24/03/2022,BSI,Annual subscription,31425
|
||||
31/03/2022,ECG FACILITIES SERVICE,Various small maintenance jobs around library sites,28760.32
|
||||
31/03/2022,XMA SCOTLAND LTD,It equipment,47461.25
|
||||
31/03/2022,JAMES BRECK LTD,Roof uplift and reslating,28230.64
|
||||
31/03/2022,LEE BOYD LIMITED,Various small maintenance jobs around library sites,26396.1
|
||||
31/03/2022,UNIVERSITY OF DUNDEE,Salary costs for SCURL Scottish Universities press project,39726.44
|
||||
30/04/2022,JISC Services Ltd,Managed router service charge annual subscription 01/04/22 to 31/03/23,25896
|
||||
30/04/2022,EX Libris,Subscription Alma and Primo 01/04/22 to 31/10/22,114420.65
|
||||
11/05/2022,KENNYS BOOKSHOP&ART GALLERIES,Purchase of Smillie Archive,30000
|
||||
12/05/2022,ECG FACILITIES SERVICE,Inspection and Maintenance of all Library properties,55711.72
|
||||
19/05/2022,CAE TECHNOLOGY SERVICES LIMITED,Subscription renewal,25041.31
|
||||
19/05/2022,GLASGOW LIFE,Kelvin Hall service charge Jan to Mar 22,59084.95
|
||||
31/05/2022,ECG FACILITIES SERVICE,Fit pre-purchased humidifiers,29710.8
|
||||
31/05/2022,ECG FACILITIES SERVICE,Routine inspection and maintenance May 22,55711.72
|
||||
31/05/2022,ALDL,Legal deposit materials April to July 22,27013.18
|
||||
09/06/2022,LEE BOYD LIMITED,Architectural Works,93690
|
||||
16/06/2022,CITY OF EDINBURGH COUNCIL,Rates for 33 Salisbury Place,136240
|
||||
16/06/2022,CITY OF EDINBURGH COUNCIL,Rates 57 George IV Bridge,41920
|
||||
23/06/2022,ECG FACILITIES SERVICE,Maintenance contract - June 22,55711.72
|
||||
21/07/2022,ALDL,"Claiming,receipting and onward distribution of UK legal deposit materials July to Sept 22",27013.16
|
||||
21/07/2022,RICK GEKOSKI,Papers 1970's to 2019 Alisdair Gray,125000
|
||||
28/07/2022,SONYA LEONARD,Literary and personal papers of Tom Leonard 1961 to 2018,40000
|
||||
|
@ -1,200 +1,200 @@
|
||||
{"text": " Morada Limited is a textile company based in Altham Lancashire. Morada specializes in curtains.", "category": "Company"}
|
||||
{"text": " The Armenian Mirror-Spectator is a newspaper published by the Baikar Association in Watertown Massachusetts.", "category": "WrittenWork"}
|
||||
{"text": " Mt. Kinka (\u91d1\u83ef\u5c71 Kinka-zan) also known as Kinkazan is located in the heart of the city of Gifu Gifu Prefecture Japan and rises to a height of 329 m (1079 ft). Previously called Mt. Inaba (\u7a32\u8449\u5c71 Inaba-yama) it has long served as the representative symbol of Gifu. It stands along the Nagara River creating bountiful nature within the city. Though it is the most famous mountain in the city Mount Dodo to the north is the tallest.", "category": "NaturalPlace"}
|
||||
{"text": " Planning the Play of a Bridge Hand is a book on contract bridge co-written by Canadian teacher and author Barbara Seagram and British author David Bird. It was published by Master Point Press in 2009.The book teaches novice bridge players some basic techniques of declarer play including suit establishment ruffing losers and the finesse.", "category": "WrittenWork"}
|
||||
{"text": " Wang Yuanping (born 8 December 1976) is a retired Chinese athlete who specialised in the 800 metres. She won several medals at the regional level.Her personal bests in the event are 2:00.63 seconds outdoors (Jinzhou 2000) and 2:03.41 seconds indoors (Yokohama 2004).", "category": "Athlete"}
|
||||
{"text": " The Incorporated VIllage of Westhampton Beach is an incorporated village in the Town of Southampton Suffolk County New York United States. As of the 2010 census the village population was 1721.", "category": "Village"}
|
||||
{"text": " Andersons Creek is a creek in Warrandyte and Park Orchards east of Melbourne Victoria Australia. It is a tributary of the Yarra River.", "category": "NaturalPlace"}
|
||||
{"text": " The Three Horseshoes is a public house in Drybridge Street in the Overmonnow area of Monmouth Wales. The pub has also been used as an Inn and also known as The Three Horse Shoes Inn. The building has been a Grade II Listed building since 15 August 1974. 19th century 2 storeys roughcast as stone with a hooded doorway", "category": "Building"}
|
||||
{"text": " The Brewer's Art is a Baltimore Maryland brewpub and restaurant. Opened on Friday September 13 1996. In 2008 it was named by Esquire magazine as the #1 Best Bar in America.", "category": "Company"}
|
||||
{"text": " The P\u00e2r\u00e2ul S\u0103r\u0103\u021bii is a tributary of the Cibin River in Romania.", "category": "NaturalPlace"}
|
||||
{"text": " Jean-Fran\u00e7ois Imbernon (born October 17 1951 in Perpignan France is a retired French international rugby union player.He played as a Lock for USA Perpignan. He earned his first cap with the French national team on 7 February 1976 against Ireland at Parc des Princes.", "category": "Athlete"}
|
||||
{"text": " Le Cadeau released in Italy as Il regalo is a 1982 French and Italian film. It stars Claudia Cardinale.", "category": "Film"}
|
||||
{"text": " Mykola Kanevets (Ukrainian: \u041c\u0438\u043a\u043e\u043b\u0430 \u041a\u0430\u043d\u0456\u0432\u0435\u0446\u044c) is the Artistic Director and Ballet Master of the Cheremosh Ukrainian Dance Company in Edmonton Alberta Canada.A native of Kiev Ukraine Mykola attended the National University of Culture and Performing Arts in Kiev Ukraine where he graduated from the Faculty of Choreography with the distinction of Ballet Master and Choreographer.", "category": "Artist"}
|
||||
{"text": " Jenna Rose Swerdlow (born September 28 1998) is an American teenage singer who gained media attention as a pre-teen with her single My Jeans. After the video went viral on YouTube and received 14 million views Swerdlow is considered a semi-viral star.", "category": "Artist"}
|
||||
{"text": " The Spice of Life is a smooth jazz studio album by Earl Klugh released in April 2008. The album received a Grammy nomination for Best Pop Instrumental Album at the 51st Grammy Awards in 2009.", "category": "Album"}
|
||||
{"text": " Lomatium macrocarpum is a perennial flowering plant in the carrot family known by the common names bigseed lomatium biscuit root or even bigseed biscuitroot. It is native to much of western North America where it can be found in various types of habitat including the grasslands of the Great Plains. It is spreading or erect perennial herb growing up to about half a meter long with hairy gray-green herbage.", "category": "Plant"}
|
||||
{"text": " Physena is the sole genus of the flowering plant family Physenaceae. It contains two species of shrubs and small trees which are endemic to Madagascar. The APG II system of 2003 (unchanged from the APG system of 1998) does recognize this family and assigns it to the order Caryophyllales in the clade core eudicots.", "category": "Plant"}
|
||||
{"text": " David John Weatherley (born 1 March 1939) is a New Zealander actor known for his roles as Spencer the butler and the voice of Benglo the Fearcat in Power Rangers Operation Overdrive and Barliman Butterbur in The Lord of the Rings: The Fellowship of the Ring.Weatherley was born in London England and moved to Canada for a military career. He eventually moved to New Zealand to engage in a theatre acting career.", "category": "Artist"}
|
||||
{"text": " Draba incrassata is an uncommon species of flowering plant in the mustard family known by the common name Sweetwater Mountains draba. It is endemic to California where it is known mainly from the Sweetwater Mountains of Mono County. It grows in alpine rock fields on the barren high mountain peaks. Draba incrassata is a small perennial herb forming mats of thick oval-shaped leaves.", "category": "Plant"}
|
||||
{"text": " Pimelea ferruginea is a small shrub native to southwest Western Australia. It was described by Labillardiere in 1805.", "category": "Plant"}
|
||||
{"text": " Lindsay Ell is a country music singer songwriter and guitarist from Calgary Alberta.She performed at the South by Southwest music festival held in Austin Texas in March 2009 the welcome reception of the 2009 Juno Awards held in Vancouver British Columbia and was also a featured artist at the 2010 Winter Olympics.", "category": "Artist"}
|
||||
{"text": " Scopula fuscata is a moth of the Geometridae family. It is found from south-western Saskatchewan west to British Columbia and south to California and Arizona. The habitat consists of montane areas including foothills.The wingspan is 24-28 mm. The wings and body are light tan sprinkled with darker yellow-brown or grey-brown scales. There is one generation per year with adults on wing in late June and early July in the northern part of the range.", "category": "Animal"}
|
||||
{"text": " Oxmoor Center is a Louisville Kentucky shopping mall located at 7900 Shelbyville Road in eastern Louisville.", "category": "Building"}
|
||||
{"text": " Ghostquake (also known as Haunted High) is a 2012 American made-for-television horror film produced by Syfy. The film was directed by Jeffrey Lando and written by Paul A. Birkett and Anthony C. Ferrante. The film stars Danny Trejo and MC Gainey. It follows a group of high school students trying to escape the wrath of a few ghastly spirits following an earthquake at their school Holloman High School.", "category": "Film"}
|
||||
{"text": " The Masonic Temple in Great Falls Montana is a building from 1914. It was listed on the National Register of Historic Places in 2000.Address is 821 Central Avenue Great Falls Motana 59401 Phone number is 453-9080.Cascade No. 34 meets 2nd and 4th Tuesdays at 7:30pm Sept-June.Euclid No. 58 meets year-round 1st and 3rd Tuesdays at 7:30pm Sept-May 3rd Tuesdays at 7:30pm June-Aug. Delta No. 128 meets 2nd Wednesdays at 7:30pm Sept-June.", "category": "Building"}
|
||||
{"text": " Harold Frederick Weaver Hawkins (1893-1977) was an English painter who specialized in ambitious sometimes mural-sized modernist allegories of morality for an age of atomic warfare and global over-population.", "category": "Artist"}
|
||||
{"text": " Robert Murray Waddington (24 October 1927 \u2013 15 March 2007) was Dean of Manchester in the last quarter of the 20th century.Born in Bognor Regis on 24 October 1927 he was educated at Dulwich College Selwyn College Cambridge and Ely Theological College. Ordained in 1954 he began his career at St John\u2019s Bethnal Green. Afterwards he was Chaplain at Slade School in Warwick Queensland. He returned to England in 1959 to join the Oratory of the Good Shepherd an order of celibate priests.", "category": "OfficeHolder"}
|
||||
{"text": " Jason Gary King (born 13 April 1985 in Maidstone England) is a speedway rider who was formerly the club captain of Newcastle Diamonds in the British Premier League. His brother Daniel is also a speedway rider.", "category": "Athlete"}
|
||||
{"text": " The African Queen is a 1951 adventure film adapted from the 1935 novel of the same name by C. S. Forester. The film was directed by John Huston and produced by Sam Spiegel and John Woolf. The screenplay was adapted by James Agee John Huston John Collier and Peter Viertel. It was photographed in Technicolor by Jack Cardiff and had a music score by Allan Gray.", "category": "Film"}
|
||||
{"text": " The Fiat Barchetta (Italian pronunciation: [\u02c8fiat bar\u02c8ketta]) (Type 183) is a roadster produced by the Italian manufacturer Fiat from 1995 to 2005 (though production was paused between May 2002 and 2004). Barchetta in Italian means 'little boat'.", "category": "MeanOfTransportation"}
|
||||
{"text": " Sardar Vallabhbhai Patel National Memorial is a museum and exhibition centre dedicated to Sardar Vallabhbhai Patel at Moti Shahi Mahal located in Shahibaug Ahmedabad Gujarat. Moti Shahi Mahal was constructed by Mughal emperor Shahjahan between 1618 and 1622. It is surrounded by a garden.", "category": "Building"}
|
||||
{"text": " Under Cover 2 is the 5th solo album of Joe Lynn Turner released in 1999. Just like Under Cover the album consists mainly of covers of Turner's favourite artists.", "category": "Album"}
|
||||
{"text": " The Atakora River is a tributary of Lake Volta in Ghana it flows about 60 km east to the Lake Volta. Its entire course is in south Ghana.", "category": "NaturalPlace"}
|
||||
{"text": " Death from Above is a 2011 horror film by director Bruce Koehler. The film features professional wrestling stars Kurt Angle Sid Eudy James Storm Matt Morgan Terry Gerin and Jessica Kresa.", "category": "Film"}
|
||||
{"text": " Portraits of Cuba is an album by Cuban musician Paquito D'Rivera released through Chesky Records in 1996. In 1997 the album won D'Rivera the Grammy Award for Best Latin Jazz Performance.", "category": "Album"}
|
||||
{"text": " Jimmy Cross (17 November 1938 - 8 October 1978) was an American radio producer and singer who attained a minor Billboard Hot 100 hit with the novelty song I Want My Baby Back in 1965. He was born in Dothan Alabama[citation needed] and became the producer of the syndicated radio series Country Concert.I Want My Baby Back was originally issued on the Tollie label and reached #92 on the Billboard Hot 100 in February 1965.", "category": "Artist"}
|
||||
{"text": " Timothy Floyd Tim Burchett (born August 25 1964) is an American Republican politician currently the mayor of Knox County Tennessee. He previously served in Tennessee General Assembly first in the Tennessee House of Representatives and later in the Tennessee State Senate in which he represented Tennessee's District 7 part of Knox County. On August 5 2010 Burchett was elected mayor of Knox County replacing Mike Ragsdale.", "category": "OfficeHolder"}
|
||||
{"text": " Daniel Lawrence Dan Whitney (born February 17 1963) best known by his stage name and character Larry the Cable Guy is an American stand-up comedian actor voice actor and former radio personality.", "category": "Artist"}
|
||||
{"text": " Renealmia is a plant genus in the family Zingiberaceae. Species include: Renealmia alpinia Renealmia aurantifera Renealmia cernua Renealmia dolichocalyx Renealmia oligotricha Renealmia sessilifolia Renealmia thrysoidesE.g. Alpinia nutans was formerly placed herein too.", "category": "Plant"}
|
||||
{"text": " Jeff Chapman (born July 17 1969 in Brunswick Georgia) is the bass singer for the Kingdom Heirs. He has been a member of the group since 2002. He has previously traveled with Bob Wills The Sound The Anchormen and The Blackwoods.He has twice been nominated as favorite bass in the Singing News fan awards.Chapman has a wife Angie two sons Justin and Sean and daughter Taylor.", "category": "Artist"}
|
||||
{"text": " Arenaria ursina is a species of flowering plant in the pink family known by the common name Bear Valley sandwort.", "category": "Plant"}
|
||||
{"text": " Living Fossil is a classic science fiction story on the concepts of human extinction and future evolution by L. Sprague de Camp. It was first published in the magazine Astounding Science-Fiction for February 1939. It first appeared in book form in the anthology A Treasury of Science Fiction (Crown Publishers 1948); it later appeared in the anthologies Gates to Tomorrow (Atheneum 1973) and The SFWA Grand Masters Volume 1 (Tor Books 1999).", "category": "WrittenWork"}
|
||||
{"text": " Brachyglottis huntii commonly called rautini or Chatham Island Christmas tree is a species in the Asteraceae family and is found only on the Chatham Islands in New Zealand.", "category": "Plant"}
|
||||
{"text": " Luktvatnet is a lake that lies in the northern part of the municipality of Vefsn in Nordland county Norway. The lake lies between the mountains Korgfjellet and Lukttinden about 5 kilometres (3.1 mi) south of Elsfjord. The European route E06 highway passes along the northern shore of the lake.", "category": "NaturalPlace"}
|
||||
{"text": " The IAR 79 is a bi-engine bomber military reconnaissance aircraft with a wood and metal structure that saw service in World War II built under licence in Brasov Romania by Industria Aeronautic\u0103 Rom\u00e2n\u0103", "category": "MeanOfTransportation"}
|
||||
{"text": " Enrico Perucconi (born 4 January 1925 in Morazzone Varese Italy) was an Italian athlete who competed mainly in the 100 metres.", "category": "Athlete"}
|
||||
{"text": " Central National-Gottesman Inc. is one of the world's largest distributors of pulp paper paperboard and newsprint. The firm's products are sold in over 75 countries through a network of 43 offices located in the United States and abroad. With annual revenues exceeding $3 billion Forbes ranked Central National-Gottesman 137th in its annual list of The Largest Private Companies.", "category": "Company"}
|
||||
{"text": " The Kout Food Group is a Kuwaiti-based conglomerate founded in 1982.In Kuwait it operates franchises of Burger King Pizza Hut and Taco Bell.Its UK arm Kout Food Group Restaurants UK it operates under brands such as Burger King KFC and Maison Blanc. In August 2013 it acquired the Little Chef chain for \u00a315 million.", "category": "Company"}
|
||||
{"text": " Fab Five: The Texas Cheerleader Scandal is a Lifetime Television made-for-TV drama film starring Jenna Dewan Ashley Benson and Tatum O'Neal and directed by Tom McLoughlin. The film premiered on August 2 2008. It is based on a true story which occurred at McKinney North High School in McKinney Texas in 2006 in which five teenage cheerleaders became notorious for bullying truancies violations of the school dress code and general disrespect to the school community and authority.", "category": "Film"}
|
||||
{"text": " Qadi Mahalleh (Persian: \u0642\u0627\u062f\u064a \u0645\u062d\u0644\u0647\u200e also Romanized as Q\u0101d\u012b Ma\u1e29alleh) is a village in Pazevar Rural District Rudbast District Babolsar County Mazandaran Province Iran. At the 2006 census its population was 228 in 59 families.", "category": "Village"}
|
||||
{"text": " Eungella Dam is one of Queensland's more established freshwater fisheries. Eungella has made a name for producing extra oversized Sooty grunter and more recently Barramundi.Eungella Dam was constructed in 1969 to meet the requirements of a thermal power station at Collinsville and the town water requirement of Collinsville and Scottsville.", "category": "NaturalPlace"}
|
||||
{"text": " The American Motor Car Company was a short-lived company in the automotive industry founded in 1906 lasting until 1913. It was based in Indianapolis Indiana United States. The American Motor Car Company pioneered the underslung design.", "category": "Company"}
|
||||
{"text": " Hawkeye & Mockingbird was a comic book ongoing series published by Marvel Comics starring superheroes Hawkeye and Mockingbird.", "category": "WrittenWork"}
|
||||
{"text": " Margaret Anderson Kelliher (born March 11 1968) is a Minnesota politician and a former member of the Minnesota House of Representatives. A member of the Minnesota Democratic\u2013Farmer\u2013Labor Party she represented District 60A which includes portions of the city of Minneapolis in Hennepin County located in the Twin Cities metropolitan area. First elected in 1999 she served until 2011 also serving as the Speaker from 2007 to 2011.", "category": "OfficeHolder"}
|
||||
{"text": " John Whitlow Wyatt (September 27 1907 \u2013 July 16 1999) was a professional baseball pitcher. He played all or part of sixteen seasons in Major League Baseball for the Detroit Tigers (1929\u201333) Chicago White Sox (1933\u201336) Cleveland Indians (1937) Brooklyn Dodgers (1939\u201344) and Philadelphia Phillies (1945). While injuries sidetracked much of Wyatt's early career he is most famous for his performance in 1941 when his team (the Dodgers) won the National League pennant.", "category": "Athlete"}
|
||||
{"text": " William Thomas Burton (31 January 1878 in Black Rock St Michael Barbados \u2013 22 August 1946 St Michael Barbados) was a coloured West Indian cricketer best known as a member of the 1900 and 1906 West Indian tourists to England. He is generally known as Tommie Burton.He was the son of a black mother and a white father. He was brought up in Barbados and served for some years there as a practice bowler and in trial matches.", "category": "Athlete"}
|
||||
{"text": " Tulemalu Lake is a lake in Kivalliq Region Nunavut Canada.", "category": "NaturalPlace"}
|
||||
{"text": " Sten Stjernqvist is a Swedish former footballer who played as a forward.", "category": "Athlete"}
|
||||
{"text": " David Parlett (born 1939) is a games scholar from South London who has studied both card games and board games. His published works include many popular books on games and the more academic volumes The Oxford Guide to Card Games and The Oxford History of Board Games both now out of print. Parlett also invented a number of board games the most successful of which is Hare and Tortoise (1974). The German edition was awarded Spiel des Jahres (Game of the Year) in 1979.", "category": "Artist"}
|
||||
{"text": " Karl Nabersberg (sometimes written as Carl Nabersberg) was a German youth leader.Nabersberg was the son of a Crefeld shopkeeper. In 1923 he joined the Jugendorganisation the forerunner of the Hitler Youth in his home town. On 28 December 1925 he was admitted as a member of the National Socialist German Workers' Party (member number 26269) and as a member of the Sturmabteilung.", "category": "OfficeHolder"}
|
||||
{"text": " \u0160etonje is a village situated in Petrovac na Mlavi municipality in Serbia.", "category": "Village"}
|
||||
{"text": " Dr. Joseph de Graft-Johnson (1933\u20131999) was an engineer academic and politician. He became the Vice-President of Ghana between 1979 and 1981.", "category": "OfficeHolder"}
|
||||
{"text": " Patties Foods (previously Patties Bakery) is an Australian food manufacturing company that produces meat pies baked goods frozen fruits and pre-made desserts. Headquartered in Bairnsdale Victoria Australia Patties Foods is represented in the Australian market by the Four'N Twenty Patties Herbert Adams Creative Gourmet Nanna's and Chefs Pride brands. Patties is the largest meat pie producing company in Australia and the world.", "category": "Company"}
|
||||
{"text": " Double Butte is the 2579-foot (786 m) mountain summit distinguished by two buttes (the other at abou 2480 feet or 756 metres) in Riverside County California. It is the western most summit of a mountain range north of Winchester California east of Perris Valley and west of the San Jacinto Valley. The eastern ridge is composed primarily of metamorphic rock of the Triassic - Jurassic French Valley formation.", "category": "NaturalPlace"}
|
||||
{"text": " Mount Carmel \u2013 Blytheswood Public School is an elementary school in the north end of Leamington Ontario Canada. It is part of the Greater Essex County District School Board and serves students from JK to Grade 8 from the communities of Blytheswood and Mount Carmel and surrounding areas.", "category": "EducationalInstitution"}
|
||||
{"text": " La combi asesina (The Killer Combination) is a 1982 Mexican film. It was directed by Gustavo Alatriste.", "category": "Film"}
|
||||
{"text": " Halimium ocymoides (basil-leaved rock rose) syn. Cistus algarvensis is a species of flowering plant in the family Cistaceae native to southern Portugal and southern Spain. It is an erect evergreen shrub growing to 60 cm (24 in) tall by 100 cm (3 ft) wide with woolly grey-green leaves and bright yellow flowers in spring. The flowers may have a dark brown blotch at the base of each petal.In cultivation this plant requires a sandy soil and full sun.", "category": "Plant"}
|
||||
{"text": " Kaala Patthar (English: Black Stone) is a 1979 Indian Bollywood action/drama film. It was produced and directed by Yash Chopra. The story was written by Salim-Javed. This film is the fourth collaboration between Amitabh Bachchan Shashi Kapoor and director Yash Chopra after the hugely successful Deewaar (1975) Kabhie Kabhie (1976) and Trishul (1978). However this film did average business at the box office. It was nominated for Filmfare awards.", "category": "Film"}
|
||||
{"text": " Martin G.S. Mansergh (born 31 December 1946) is a former Irish Fianna F\u00e1il politician and historian. He was a Teachta D\u00e1la (TD) for the Tipperary South constituency from 2007 until 2011. He was previously a Senator from 2002 to 2007. He played a leading role in formulating Fianna F\u00e1il policy on Northern Ireland.", "category": "OfficeHolder"}
|
||||
{"text": " Shriniwas Ganesh Sardesai (1907-1996) popularly known as S.G. Sardesai was an Indian freedom fighter from Maharashtra and one of the great communist leaders produced by the communist movement in India. He is author of the book Progress and conservatism in ancient India famous for his profound theoretical analysis. He was the Central Executive Committee of pre-split Communist Party of India during the Indo-china conflict.", "category": "OfficeHolder"}
|
||||
{"text": " USS Tuluran (AG-46) \u2013 also known as USS Lake Superior (ID-2995) \u2013 was a commercial cargo ship acquired by the U.S. Navy for service during both World War I when she was known as USS Lake Superior and also during World War II when she was known as USS Tuluran.", "category": "MeanOfTransportation"}
|
||||
{"text": " The American Journal of Gastroenterology is a peer-reviewed medical journal published for the American College of Gastroenterology by the Nature Publishing Group.", "category": "WrittenWork"}
|
||||
{"text": " William Lindsay (September 4 1835 \u2013 October 15 1909) was a Democratic U.S. Senator from Kentucky from 1893 to 1901.Born near Lexington Virginia Lindsay attended the common schools and settled in Clinton Kentucky in 1854. There he taught school and studied law. He was admitted to the bar and commenced practice in Clinton in 1858.", "category": "OfficeHolder"}
|
||||
{"text": " Brian Schroeder (a.k.a. Pushead) is an artist record label owner and writer within the hardcore punk and heavy metal field. He has created artwork for many bands artists and athletes including Metallica The Misfits Dr. Dre Travis Barker Craig Johnson and Kool Keith. He has designed many record covers T-shirts skateboards and a pair of Nike SB Dunks. His record label Pusmort Records has released albums by Negative Gain Poison Idea and Final Conflict.", "category": "Artist"}
|
||||
{"text": " Panicum anceps is a species of grass known by the common name beaked panicgrass. It is native to the southeastern United States where it occurs as far north as New Jersey and as far west as Kansas and Texas.This species is a rhizomatous perennial grass with stems growing up to 1.3 meters tall. The leaves have erect blades up to half a meter tall. The inflorescence is a panicle up to 40 centimeters long bearing pale green or yellowish spikelets. The grass produces an abundance of seed.", "category": "Plant"}
|
||||
{"text": " Shukan ST is a weekly newspaper published by The Japan Times for learners of English language. It is originally titled as Student Times but changed to Shukan ST since a significant portion of its readers are not students. It has articles on news movie lifestyle in English-speaking countries opinions and other kinds attracting learners of English and helping them with notes on terms.", "category": "Company"}
|
||||
{"text": " The Tiger Hotel is a hotel in Columbia Missouri. Built as a hotel in 1928 the building later housed a retirement home and banquet center. In 2012 the building was fully restored and reopened as a boutique hotel. It was listed on the National Register of Historic Places in 1980.", "category": "Building"}
|
||||
{"text": " Emi Motoi (\u672c\u4e95 \u3048\u307f Motoi Emi born October 11 in Kanagawa) is a Japanese voice actress.", "category": "Artist"}
|
||||
{"text": " The Hudson River is a 49.5-mile-long (79.7 km) tributary of the Broad River in the U.S. state of Georgia. Via the Broad River it is part of the Savannah River watershed.The headwaters are in Banks County near the city of Homer. Grove Creek feeds into the Hudson near the Franklin County line. The river then constitutes most of the southern border of Franklin County separating it from Madison County.", "category": "NaturalPlace"}
|
||||
{"text": " This article details Car Nos. 10\u201313 of the Manx Electric Railway on the Isle of Man.This was the third batch of motorcars delivered to the railway in 1895 at the same time as the cars for the new Snaefell Mountain Railway were delivered. They were constructed to a very similar design to those provided for the mountain line.", "category": "MeanOfTransportation"}
|
||||
{"text": " Catharanthus roseus commonly known as the Madagascar rosy periwinkle is a species of Catharanthus native and endemic to Madagascar. Other English names occasionally used include Cape periwinkle rose periwinkle rosy periwinkle and old-maid.", "category": "Plant"}
|
||||
{"text": " Thapanzeik is a village in Homalin Township Hkamti District in the Sagaing Region of northwestern Burma.", "category": "Village"}
|
||||
{"text": " USS Spiegel Grove (LSD-32) was a Thomaston-class dock landing ship of the United States Navy. She was named for Spiegel Grove the home and estate in Fremont Ohio of Rutherford B. Hayes the 19th President of the United States.", "category": "MeanOfTransportation"}
|
||||
{"text": " Acmella is a genus of thirty species of plants in the aster family Asteraceae. It is native to the Americas and has been introduced to Asia Africa the Pacific islands and Australia.One familiar species is Acmella oleracea which has been widely cultivated for centuries. It is used for food and medicine and as an insecticide and an ornamental plant.", "category": "Plant"}
|
||||
{"text": " Mirbelia is a plant genus belonging to the Fabaceae family. It is endemic to Australia occurring in every mainland state except South Australia.", "category": "Plant"}
|
||||
{"text": " Nigma puella is a species of spider belonging to the family Dictynidae. It is found in Europe Azores Madeira Canary Islands and parts of North Africa.Like most members of the family this is a small spider but the female is striking with a light green abdomen marked with a bold maroon blotch and a variable amount of barring in the same colour. The male is reddish-brown. This species makes a horizontal web over the top surface of a leaf.", "category": "Animal"}
|
||||
{"text": " The Madrisa (or Madrisahorn) is a mountain in the R\u00e4tikon mountain range overlooking Klosters in the Swiss canton of Graub\u00fcnden. Its summit (2826 metres) is located near the Austrian border.The Madrisa is constituted by several secondary summits notably the Gargeller Madrisa (2770 metres) overlooking Gargellen in Austria.Ski lifts up to 2600 metres are located on the Klosters side.", "category": "NaturalPlace"}
|
||||
{"text": " Temporary Temple is a live album by Psychic TV. The album was recorded on July 28 1984 in London and released on 12 vinyl. It was later coupled with another concert and released on CD as Temporary Temple & Atonal.", "category": "Album"}
|
||||
{"text": " La hija de Juan Sim\u00f3n (Juan Sim\u00f3n's Daughter) is a musical play by Nemesio M. Sobrevila which has been made into two Spanish films. It is also the name of the title track and the song has been recorded by numerous artists such as Leonardo Favio.The first film directed by Jos\u00e9 Luis S\u00e1enz de Heredia was released in 1935 and starred Angelillo Pilar Mu\u00f1oz and Manuel Arb\u00f3. Luis Bu\u00f1uel was the executive producer for Film\u00f3fono and had a small role as an actor.", "category": "Film"}
|
||||
{"text": " Book Of Matches is a poetry book written by Simon Armitage first published in 1993 by Faber and Faber. Several poems featured in the book are studied as part of the GCSE English Literature examination in the UK.The book is written in three sections the first (Book of Matches) containing 30 short sonnets. Each is meant to be read within 20 seconds the amount of time it would take for a match to be lit and burn out.", "category": "WrittenWork"}
|
||||
{"text": " The Last Supper is the fourth album released by American stand-up comedian Jim Gaffigan. It focuses largely on his love of food.", "category": "Album"}
|
||||
{"text": " The Miami Center is a skyscraper in downtown Miami Florida. Although Miami Center is not the city's tallest building it is a symbol of early downtown. Built in 1983 it is older compared with most of the taller buildings in Miami which have been built in the last decade. In addition the Miami Center is immediately adjacent to Bayfront Park and is unobstructed when looking at the skyline from Miami Beach to the east. The building is 484 ft (148 m) tall and has 34 floors.", "category": "Building"}
|
||||
{"text": " Duboisia hopwoodii is a shrub native to the arid interior region of Australia. Common names include pituri pitchuri thornapple or pitcheri. It has an erect habit usually growing to between 1 and 3 metres in height and has long narrow leaves. Flowers are white and bell-shaped with violet-striped throats. These appear between June and November in the species native range followed by purple-black rounded berries which are 3 to 6 mm in diameter.", "category": "Plant"}
|
||||
{"text": " Jelenin svet (Jelena's World) is a 2008 independent documentary film written and directed by Tanja Brzakovi\u0107 about former World No. 1 female tennis player Jelena Jankovi\u0107.", "category": "Film"}
|
||||
{"text": " Jay Cashman Inc. is an American heavy-construction company based in Quincy Massachusetts with satellite offices in Boston Jupiter Florida and Staten Island New York. As of 2006 the company has about 1000 employees. The company was one of the major contractors on the Boston's Central Artery/Tunnel Project. In 2004 Jay Cashman Inc.", "category": "Company"}
|
||||
{"text": " Hashemanli (Persian: \u0647\u0627\u0634\u0645\u0646\u0644\u064a\u200e also Romanized as H\u0101shemanl\u012b; also known as H\u0101shem El\u00e1) is a village in Jafarbay-ye Jonubi Rural District in the Central District of Torkaman County Golestan Province Iran. At the 2006 census its population was 135 in 27 families.", "category": "Village"}
|
||||
{"text": " Rani Kasula Rangamma is a Telugu film starring Chiranjeevi.", "category": "Film"}
|
||||
{"text": " The 20/20 Experience \u2013 The Complete Experience is a compilation album by American singer-songwriter Justin Timberlake. It was released on September 27 2013 by RCA Records.", "category": "Album"}
|
||||
{"text": " R.C. Bigelow Inc better known as the Bigelow Tea Company is an American tea company based in Fairfield Connecticut. The company was founded by Ruth C. Bigelow in the late 1940s based on a recipe she marketed as Constant Comment tea. Bigelow is still a 100% family-owned business that markets over 50 varieties of tea including black and green as well as herbal teas all of which are still blended in Fairfield. They also own America's only tea plantation in Charleston South Carolina.", "category": "Company"}
|
||||
{"text": " Thomas Eyre(fl. 1890s) was a footballer who made 65 appearances in the Football League playing for Lincoln City. He played at left back. Either side of Lincoln he played for Ashfield and Hamilton Academical in Scotland.", "category": "Athlete"}
|
||||
{"text": " Malleable Iron Range Company was a company that existed from 1896 to 1985 and primarily produced kitchen ranges made of malleable iron but also produced a variety of other related products. The company's primary trademark was 'Monarch' and was colloquially often referred to as the Monarch Company or just Monarch.", "category": "Company"}
|
||||
{"text": " The Chiltern School is a coeducational special school located over two sites in Dunstable and Houghton Regis in Bedfordshire England. The school accepts pupils from all over the Central Bedfordshire area.The school was formed in 2012 from the merger of Glenwood School in Dunstable and Hillcrest School in Houghton Regis.", "category": "EducationalInstitution"}
|
||||
{"text": " Kim Dae-Eun (born September 17 1984) is a South Korean gymnast. He is the 2004 Olympic All-around silver medalist. He won the gold medal on the parallel bars at the 2007 World Artistic Gymnastics Championships.Kim was part of the South Korean team that won the bronze medal in the team event at the 2006 Asian Games.", "category": "Athlete"}
|
||||
{"text": " Arayik Vladimirovich Harutyunyan (Armenian: \u0531\u0580\u0561\u0575\u056b\u056f \u0540\u0561\u0580\u0578\u0582\u0569\u0575\u0578\u0582\u0576\u0575\u0561\u0576 Russian: \u0410\u0440\u0430\u0438\u043a \u0410\u0440\u0443\u0442\u044e\u043d\u044f\u043d) (born 14 December 1973) is the current Prime Minister of the Nagorno-Karabakh Republic. He was suggested by the President of Nagorno-Karabakh Bako Sahakyan and was unanimously approved by the Parliament of Karabakh on 14 September 2007 by 32 out of 32 present parliamentarians.", "category": "OfficeHolder"}
|
||||
{"text": " Shelton Hank Williams also known as Hank Williams III and Hank 3 (born December 12 1972) is an American musician singer and multi-instrumentalist including guitar bass drums banjo and vocals. In addition to his honky tonk recordings Williams' style alternates between country punk and metal.", "category": "Artist"}
|
||||
{"text": " Helicella orzai is a species of air-breathing land snails terrestrial pulmonate gastropod mollusks in the family Hygromiidae the hairy snails and their allies. This species is endemic to Spain.", "category": "Animal"}
|
||||
{"text": " Gro\u00dfe Schmalenau is a river of North Rhine-Westphalia Germany.", "category": "NaturalPlace"}
|
||||
{"text": " The Tupolev ANT-29 (military designation DIP \u2013 Dvukhmotorny istrebitel pushechny twin-engined cannon fighter) was a 1930s twin-engined cannon-armed fighter designed by Alexander Arkhangelsky and built by Tupolev.Design work started in 1932 on a twin-engined aircraft capable of carrying two APK-100 cannons. The resulting design was the ANT-29 and it first flew in February 1935. A monoplane with a tall and narrow fuselage and powered by two Hispano-Suiza 12Ybrs engines.", "category": "MeanOfTransportation"}
|
||||
{"text": " Charles Corm (1894-1963) was a Lebanese writer businessman and philanthropist. He is considered to be the leader of the Phoenicianism movement in Lebanon which ignited a surge of nationalism that led to Lebanon's independence. In a country torn by sectarian conflicts Corm's intention was to find a common root shared by all Lebanese beyond their religious beliefs (the Phoenicians were pagans).", "category": "Artist"}
|
||||
{"text": " Joseph Hubert Ruetz (October 21 1916 \u2013 January 2 2003) was a professional football player in the All-America Football Conference for the Chicago Rockets in 1946 and 1948. Prior to that he played at the collegiate level while attending the University of Notre Dame. He played guard for the Irish with the exception of playing one season at quarterback. In 1938 he graduated from Notre Dame with cum laude honors.", "category": "Athlete"}
|
||||
{"text": " The Reef House is a historic house located at 411 S. Poplar St. in Carbondale Illinois. William A. Reef built the house for his family circa 1892. The Queen Anne-style cottage may have been designed by local carpenter A. M. Etherton though records of its designer do not exist. The house features fishscale shingle siding on its second floor and clapboard siding on its first; the clapboard siding is adorned with stickwork.", "category": "Building"}
|
||||
{"text": " MAKO Surgical Corp. (Stryker Medical) is a publicly traded medical device company based in Florida. On September 25 2013 the Board of Directors of Mako Surgical accepted a deal to merge with Stryker Medical for $1.65B subject to shareholder approval.", "category": "Company"}
|
||||
{"text": " Pop Carn is a 2003 Indian Tamil film written and directed by actor-cum-director Nassar and starring Mohanlal and Simran Bagga in lead roles and introducing newcomers Kunal Shah and Jyothi Nawal. The film which had music scored by Yuvan Shankar Raja was released on 30 January 2003 but flopped at the box office. Nonetheless the film was dubbed into Malayalam and released in 2007 under the same name.", "category": "Film"}
|
||||
{"text": " USNS Mount Baker (T-AE-34) is the seventh of eight Kilauea-class ammunition ships to serve with the Military Sealift Command. She is the second U.S. Navy ship to bear the name and is named for Mount Baker a 10781-foot volcano in the Cascade Range of Washington. Ammunition ships operated by Military Sealift Command provide logistic support to US Navy ships at sea.Mount Baker was built by Ingalls Shipbuilding Pascagoula Mississippi.", "category": "MeanOfTransportation"}
|
||||
{"text": " Dansere is an album by Jan Garbarek. The album was recorded in November 1975 and features the Bobo Stenson Quartet.", "category": "Album"}
|
||||
{"text": " Divraz (Persian: \u062f\u064a\u0648\u0631\u0632\u200e also Romanized as D\u012bvraz) is a village in Bala Khiyaban-e Litkuh Rural District in the Central District of Amol County Mazandaran Province Iran. At the 2006 census its population was 393 in 95 families.", "category": "Village"}
|
||||
{"text": " The D\u0103ih\u0103\u021ba\u0219u River is a tributary of the Dumbr\u0103vanu River in Romania.", "category": "NaturalPlace"}
|
||||
{"text": " Zeisters also known as Fat Guy Goes Nutzoid is a 1986 comedy film produced by Troma Entertainment. Troma was originally set to title the film Fat Boy Goes Nutzoid but at the request of the lawyers of the hip-hop group The Fat Boys it was changed to Fat Guy.", "category": "Film"}
|
||||
{"text": " Paul Gobeil (born March 1 1942 in Saint-R\u00e9mi-de-Tingwick Quebec) is a former Canadian politician and businessman.From 1985 to 1989 Mr. Gobeil was a Liberal member of the National Assembly for the riding of Verdun and served as Minister assigned to Administration President of the Treasury Board and as Minister of International Affairs for the Government of Quebec.", "category": "OfficeHolder"}
|
||||
{"text": " Ruff Ryders: Past Present Future is the fifth compilation album from American hip hop record label Ruff Ryders Entertainment released on November 21 2011.", "category": "Album"}
|
||||
{"text": " Ridi Viharaya (Sinhala: \u0dbb\u0dd2\u0daf\u0dd3 \u0dc0\u0dd2\u0dc4\u0dcf\u0dbb\u0dba) or Silver Temple is a 2nd-century BCE Theravada Buddhist temple in the village of Ridigama Sri Lanka. Built during the reign of Dutthagamani of Anuradhapura the temple is considered as the place where the silver ore which provided silver to complete Ruwanwelisaya; one of the largest stupa in Sri Lanka was discovered.", "category": "Building"}
|
||||
{"text": " Grand Canyon Preparatory Academy is a public charter college preparatory school in Tempe Arizona.", "category": "EducationalInstitution"}
|
||||
{"text": " Aricoceras is an extinct genus of the Adrianitidae family. They are an extinct group of ammonoid which are shelled cephalopods related to squids belemnites octopuses and cuttlefish and more distantly to the nautiloids.", "category": "Animal"}
|
||||
{"text": " Blackburn High School is a public secondary school for girls and boys in years 7 to 12 in Blackburn a suburb of Melbourne Victoria Australia. Blackburn High School is an outstanding secondary school for aspiring young men and women. It aims to educate tomorrow's minds today. Started in 1956 the school has a proud tradition of academic excellence and exceptional music achievement.The school is nationally recognised as a leading educational institution for music education.", "category": "EducationalInstitution"}
|
||||
{"text": " Chris Nieratko (born February 19 1976) is an American humorist and author. Nieratko is a past editor of Big Brother Magazine and currently reviews pornographic films for Vice magazine as well as being the author of the related Skinema book. He also appeared on MTV's Jackass.", "category": "Artist"}
|
||||
{"text": " Warlock is a 1959 film released by Twentieth Century Fox and shot in DeLuxe Color and CinemaScope. It is a Western adapted from the novel by Oakley Hall (screenplay written by Robert Alan Aurthur).", "category": "Film"}
|
||||
{"text": " Sieniawa [\u0255e\u02c8\u0272ava] is a village in the administrative district of Gmina Raba Wy\u017cna within Nowy Targ County Lesser Poland Voivodeship in southern Poland. It lies approximately 5 kilometres (3 mi) south-east of Raba Wy\u017cna 10 km (6 mi) north-west of Nowy Targ and 59 km (37 mi) south of the regional capital Krak\u00f3w.The village has a population of 1900.", "category": "Village"}
|
||||
{"text": " Michael Adam (born 9 December 1984) is a German politician. He has been District Administrator (Landrat) of Regen since 2011.", "category": "OfficeHolder"}
|
||||
{"text": " Thunderbird High School is a public high school located in northwestern Phoenix Arizona. The school is a part of the Glendale Union High School District.", "category": "EducationalInstitution"}
|
||||
{"text": " Nayef Al Khater (born May 10 1978) is a Qatari football player. He currently plays for Al Wakrah as a defender.", "category": "Athlete"}
|
||||
{"text": " Black Cobra Woman (Italian: Eva nera) also known as Black Cobra is an Italian 1976 exploitation movie written and directed by Joe D'Amato.", "category": "Film"}
|
||||
{"text": " Joe Cuba a.k.a Sonny (April 22 1931 \u2013 February 15 2009) was a musician of Puerto Rican descent who was known as the Father of Latin Boogaloo.", "category": "Artist"}
|
||||
{"text": " Jacob LeBlanc (born February 2 1981 in Auburn California) is an American retired professional soccer player.", "category": "Athlete"}
|
||||
{"text": " Kevin B. Kamenetz is the 12th and current County Executive of Baltimore County Maryland serving since 2010. He is a member of the Democratic Party. He previously served as a four-term County Councilman representing the Second District of Baltimore County.", "category": "OfficeHolder"}
|
||||
{"text": " Thomas Frederick Fred Peart Baron Peart PC (30 April 1914 \u2013 26 August 1988) was a British Labour politician who served in the Labour governments of the 1960s and 1970s and was a candidate for Deputy Leader of the Party.", "category": "OfficeHolder"}
|
||||
{"text": " Grand Lake is a large lake in the interior of Newfoundland of the Canadian province of Newfoundland and Labrador. It has an area of 534 km\u00b2 making it the largest lake on Newfoundland. Consequently it is one of if not the deepest.", "category": "NaturalPlace"}
|
||||
{"text": " A Colossal Failure of Common Sense: The Inside Story of the Collapse of Lehman Brothers is a 2009 non-fiction book written by Lawrence G. McDonald and Patrick Robinson which chronicles the events surrounding the bankruptcy of Lehman Brothers in the context of the financial crisis of 2007\u20132010 and the subprime mortgage crisis.", "category": "WrittenWork"}
|
||||
{"text": " Thatching (31 May 1975 \u2013 1999) was an Irish Thoroughbred racehorse and sire. The horse's early career was delayed and disrupted by injury and he did not show his best form until switched to sprinting distances in the spring of 1979 when he won the Duke of York Stakes. He improved further when equipped with blinkers that summer recording impressive victories in both the Cork and Orrery Stakes and the July Cup.", "category": "Animal"}
|
||||
{"text": " Ya\u015far Kurt (Armenian: \u0545\u0561\u0577\u0561\u0580 \u053f\u0578\u0582\u0580\u0569 b.August 16 1968 in Istanbul Turkey) is a Turkish-Armenian rock artist.", "category": "Artist"}
|
||||
{"text": " Then and Now is a historical novel by W. Somerset Maugham. Set in Florence Italy during the Renaissance the story focuses on three months in the life of Niccolo Machiavelli the Florentine politician diplomat philosopher and writer in the early years of the 16th century. The book was first published by Heinemann in 1946.", "category": "WrittenWork"}
|
||||
{"text": " Abdollah Masud-e Sofla (Persian: \u0639\u0628\u062f\u0627\u0644\u0647 \u0645\u0633\u0639\u0648\u062f\u0633\u0641\u0644\u064a\u200e also Romanized as \u2018Abdoll\u0101h Mas\u2018\u016bd-e Sofl\u00e1; also known as Abdollah Mas\u2019ood \u2018Abdoll\u0101h Mas\u2018\u016bd and Abdull\u0101h Mas\u016bd) is a village in Hesar-e Valiyeasr Rural District Avaj District Buin Zahra County Qazvin Province Iran. At the 2006 census its population was 72 in 15 families.", "category": "Village"}
|
||||
{"text": " Springhill High School (SHS) is a secondary school in Springhill Nova Scotia Canada. SHS is part of the Chignecto-Central Regional School Board and is the only high school in the town of Springhill. The school is home to many sports teams and clubs. These include: basketball soccer badminton track and field softball Students Against Destructive Decisions drama club homework club and book club.", "category": "EducationalInstitution"}
|
||||
{"text": " Charniele L. Herring (/\u0283\u0251r\u02c8n\u025bl \u02c8h\u025br\u026a\u014b/ shar-NEL HERR-ing; born September 25 1969) is an American politician. She has served in the Virginia House of Delegates since 2009 representing the 46th district made up the city of Alexandria and part of Fairfax County near Washington D.C. Herring is a member of the Democratic Party. She has been the House minority whip since 2012 and in December 2012 she was the first African-American to be elected chair of the Democratic Party of Virginia.", "category": "OfficeHolder"}
|
||||
{"text": " Symmoca dodecatella is a moth of the Symmocidae family. It is found in Portugal and Spain.The wingspan is about 18\u201319 mm. The forewings are grey sprinkled with black mainly along the margin. The hindwings are grey.", "category": "Animal"}
|
||||
{"text": " Ali Abbasov Mammad oglu (Azerbaijani: \u018fli Abbasov M\u0259mm\u0259d o\u011flu) (born 1953 in Azerbaijan) is the current Minister of Communications and Information Technologies of the Republic of Azerbaijan.", "category": "OfficeHolder"}
|
||||
{"text": " Worlds Beyond was an American digest magazine of science fiction and fantasy fiction in 1950 and 1951. The magazine only issued three monthly issues from December 1950 to February 1951 but is notable for having printed stories by Cyril M. Kornbluth Jack Vance Mack Reynolds Graham Greene John Christopher Lester del Rey Judith Merril and others.Worlds Beyond was published by Hillman Periodicals and was edited by Damon Knight.", "category": "Company"}
|
||||
{"text": " The Daily News Journal commonly abbreviated to DNJ is a newspaper serving Murfreesboro Tennessee Rutherford County and surrounding communities. Published in Murfreesboro it serves as the primary local newspaper with competition from The Murfreesboro Post and other publications. The newspaper is not in competition with The Tennessean of Nashville as both are owned by Gannett Company.The roots the DNJ date back to 1849 and the founding of Murfreesboro News.", "category": "WrittenWork"}
|
||||
{"text": " Echinocereus fendleri is a species of cactus known by the common names pinkflower hedgehog cactus and Fendler's hedgehog cactus. It grows in deserts and woodlands in the Southwestern United States and Northeastern Mexico. It is most common in New Mexico.The taxonomy of the species is uncertain with authors recognizing up to eight varieties.", "category": "Plant"}
|
||||
{"text": " Michael F. Kitt Snr (13 September 1914 \u2013 24 December 1974) was an Irish Fianna F\u00e1il politician and long-serving Teachta D\u00e1la (TD).He was elected to D\u00e1il \u00c9ireann for the first time at the 1948 general election for the Galway North constituency but lost his seat at the 1951 general election and failed to be elected again at the 1954 general election.", "category": "OfficeHolder"}
|
||||
{"text": " Epidendrum mancum is an epiphytic orchid that grows in the tropical low elfin cloud forests of Ecuador and Amazonas Peru at altitudes of 2\u20143 km .", "category": "Plant"}
|
||||
{"text": " Salempur Masanda is a village in Jalandhar District near the Jalandhar Cantonment in Punjab India.", "category": "Village"}
|
||||
{"text": " Yaleh Gonbad (Persian: \u064a\u0644\u0647 \u06af\u0646\u0628\u062f\u200e; also known as Em\u0101mz\u0101deh Imamzade-Ele-Geumbez and Im\u0101mz\u0101deh) is a village in Ilat-e Qaqazan-e Gharbi Rural District Kuhin District Qazvin County Qazvin Province Iran. At the 2006 census its population was 429 in 105 families.", "category": "Village"}
|
||||
{"text": " Popeyes Louisiana Kitchen is an American chain of fried chicken fast food restaurants founded in 1972 in New Orleans Louisiana. Often referred to as Popeyes and sometimes as Popeyes Chicken & Biscuits or Popeyes Chicken & Seafood[citation needed] it was acquired by Sandy Springs Georgia-based AFC Enterprises originally America's Favorite Chicken Company in 1993.", "category": "Company"}
|
||||
{"text": " The White Umfolozi River originates just south of Vryheid KwaZulu-Natal South Africa and joins the Black Umfolozi River at 28\u00b020\u203258\u2033S 31\u00b058\u203246\u2033E to form the Umfolozi River before it flows east towards the Indian Ocean.", "category": "NaturalPlace"}
|
||||
{"text": " The Albatros L 74 was a two-seated German training biplane produced by Albatros Flugzeugwerke. Only two were produced.", "category": "MeanOfTransportation"}
|
||||
{"text": " The University of Nevada School of Medicine is an academic division of the University of Nevada Reno and grants the Doctor of Medicine (MD) degree. The School of Medicine was founded in 1969 as the first medical school in the state of Nevada. More than 1500 MDs have graduated from the School of Medicine. The pre-clinical campus is located in Reno but the third and fourth years can be spent in hospitals and clinics throughout Nevada.", "category": "EducationalInstitution"}
|
||||
{"text": " Leon Kroll (December 6 1884 \u2013 October 25 1974) was an American painter and lithographer. Known as a figurative artist Life Magazine described him as the dean of U.S. nude painters yet he was an exceptional landscape painter and also produced an exceptional body of still life compositions.Born into a musical family on lower Second Avenue in New York City Kroll's father was a violinist and his cousin was William Kroll.", "category": "Artist"}
|
||||
{"text": " Michael E. DeBakey High School for Health Professions at Qatar (DHSHP@Q in short) is a private international middle and secondary school in Doha Qatar. The school is a branch campus of Michael E. DeBakey High School for Health Professions of Houston Texas United States. Charlesetta Deason is the CEO and President.Named after Michael E. DeBakey the school opened in September 2008 with grades 8 through 10 with 100 students per grade; the school will ultimately cover grades 7-12.", "category": "EducationalInstitution"}
|
||||
{"text": " The Richleighs of Tantamount is a children\u2019s historical novel written by British author Barbara Willard. It was originally published in the United Kingdom in 1966 by the publishers Constable before being published in the United States by Harcourt Brace & World in June 1967. C. Walter Hodges drew the line illustrations and painted the cover portrait for the original edition.", "category": "WrittenWork"}
|
||||
{"text": " Ennea is a genus of air-breathing land snails terrestrial pulmonate gastropod mollusks in the family Streptaxidae.Ennea is the type genus of the subfamily Enneinae.", "category": "Animal"}
|
||||
{"text": " Come Live with Me is a 1941 American romantic comedy film produced and directed by Clarence Brown and starring James Stewart and Hedy Lamarr. Based on a story by Virginia Van Upp the film is about a beautiful Viennese refugee seeking United States citizenship who arranges a marriage of convenience with a struggling writer.", "category": "Film"}
|
||||
{"text": " St. Thomas Episcopal Church is a parish church in the Episcopal Diocese of Iowa. The church is located in Sioux City Iowa United States at 1200 Douglas Street. The church building is listed on the National Register of Historic Places.", "category": "Building"}
|
||||
{"text": " Nuno Daniel Costeira Valente (born 22 November 1991 in Ada\u00fafe - Braga ) is a Portuguese footballer who plays for Vizela on loan from S.C. Braga as a midfielder.", "category": "Athlete"}
|
||||
{"text": " Jaagoo (Bengali: \u099c\u09be\u0997\u09cb) is a Bangladeshi sports based romantic Movies. Its writer and director Khijir Hayat Khan. Adnan Karim sponsored youth football game built this film was released in 2010. The film is produced by Sharjeel Karim and Adnan Karim and directed by Khijir Hayat Khan who has written the story screenplay and dialogues of the film. The film features Ferdous Ahmed and Afsana Ara Bindu in lead roles and with supporting Arefin Shuvo Tariq Anam Ronok Hasaan and many more.", "category": "Film"}
|
||||
{"text": " John Edward Hatton AO (born 29 May 1933) is former Australian politician and an National Trust of Australia nominated Australian Living Treasure. He was the independent member of the Legislative Assembly of the New South Wales parliament for the seat of South Coast from 1973 to 1995. Notably the allegations about police corruption Hatton raised in Parliament resulted in the Wood Royal Commission. He is currently a social activist in his local community.", "category": "OfficeHolder"}
|
||||
{"text": " Trichoptilus subtilis is a moth of the Pterophoridae family that is known from South Africa.", "category": "Animal"}
|
||||
{"text": " Sin\u00e9ad Madden (born in Galway Ireland) is an Irish singer-songwriter and fiddle player best known as a member of the Moya Brennan band. She also teaches at Waltons New School of Music in Dublin.", "category": "Artist"}
|
||||
{"text": " Philip Sprint is a German footballer who currently plays for Hertha BSC.Sprint made his professional debut for Hertha BSC on 12 August 2012 in a 2. Bundesliga match against FSV Frankfurt coming on in the 50th minute for Marvin Knoll after starting goalkeeper Sascha Burchert had been sent off.", "category": "Athlete"}
|
||||
{"text": " River Roads Mall was an enclosed shopping mall located in the city of Jennings a suburb of St. Louis Missouri United States. Opened in 1962 as one of the nation's first shopping malls the mall declined in the 1990s becoming a dead mall and eventually being shuttered in 1995. Demolition of the long-vacant mall began in 2006.", "category": "Building"}
|
||||
{"text": " The Brown-patched Kangaroo lizard (Otocryptis wiegmanni) also called Wiegmann's Agama or Sri Lankan Kangaroo Lizard is a small ground dwelling agamid lizard endemic to the wet zone forests and lower mountain forests (up to 1300 metres) of Sri Lanka. It is commonly seen in the leaf litter of shady rain forests. When perceiving danger it spurts away quickly on its large hind legs and might eventually climb up a sapling or tree.", "category": "Animal"}
|
||||
{"text": " Shiho Kawaragi (\u6cb3\u539f\u6728 \u5fd7\u7a42 Kawaragi Shiho born April 29 1976 in Tokyo) is a Japanese voice actress who works for Kenyu-Office. When voicing adult games and hentai OVAs she is also known as Kaname Yuzuki (\u67da\u6728\u304b\u306a\u3081 Yuzuki Kaname) She is currently married since March 2012.", "category": "Artist"}
|
||||
{"text": " Down in the Shacks Where the Satellite Dishes Grow is the second album by the Judybats released in 1992.", "category": "Album"}
|
||||
{"text": " Turn of Faith is a 2001 film directed by Charles Jarrott. It stars Ray Mancini and Mia Sara.", "category": "Film"}
|
||||
{"text": " Frederick William Seward (July 8 1830 \u2013 April 25 1915) was the Assistant Secretary of State during the American Civil War serving in Abraham Lincoln's administration as well as under Andrew Johnson during Reconstruction and for over two years under Rutherford B. Hayes.", "category": "OfficeHolder"}
|
||||
{"text": " Ivoprop Corporation founded in 1984 by Ivo Zdarsky is an American manufacturer of composite propellers for homebuilt and ultralight aircraft as well as airboats. The company headquarters is located in Bellflower California.Zdarsky started the company after carving his own propeller for a homebuilt ultralight trike that he flew from Cold War Czechoslovakia over the Iron Curtain to Vienna in 1984.", "category": "Company"}
|
||||
{"text": " Wave Broadband is a provider of residential business and enterprise class cable TV broadband internet and telephone services on the West Coast currently serving about 400000 customers within communities in western Washington state Oregon Sacramento California and the San Francisco Bay Area. Wave Broadband provides services via their fiber-optic network and uses Northwest Open Access Network as the backbone for most of their service areas in Washington.", "category": "Company"}
|
||||
{"text": " Andie Tong is a comic book artist known for his work on books such as Spectacular Spider-Man UK The Batman Strikes! and Tangent: Superman's Reign.", "category": "Artist"}
|
||||
{"text": " Merdani is a village in the municipality of Busova\u010da Bosnia and Herzegovina.", "category": "Village"}
|
||||
{"text": " Kamam (Persian: \u0643\u0627\u0645\u0645\u200e also Romanized as K\u0101mam) is a village in Mangur-e Sharqi Rural District Khalifan District Mahabad County West Azerbaijan Province Iran. At the 2006 census its population was 98 in 16 families.", "category": "Village"}
|
||||
{"text": " Ficus greiffiana is a species of plant in the Moraceae family. It is found in Argentina Brazil Colombia and Guyana.", "category": "Plant"}
|
||||
{"text": " Toni Amboaje is a Spanish singer who currently works for metal band Sauze which formed on early 2008.", "category": "Artist"}
|
||||
{"text": " Mount Whittier is a mountain in Carroll County New Hampshire in the northern Ossipee Mountains. Named after John Greenleaf Whittier the peak is not to be confused with nearby Nickerson Mountain which was once known as Mount Whittier.There are no hiking trails on Mount Whittier. There was once a CCC alpine ski trail on the northern face.", "category": "NaturalPlace"}
|
||||
{"text": " El Rompe Discoteka: The Mix Album is a 2007 album by Hector El Father.", "category": "Album"}
|
||||
{"text": " e-Spirit is a commercial software company that develops and markets the FirstSpirit CMS Web content management system. The company was founded in 1999 in Dortmund Germany and established a US presence in 2011. The company's FirstSpirit CMS is a Java-based offering now in its fifth major release.[citation needed]", "category": "Company"}
|
||||
{"text": " The Valley is the first novel by Barry Pilton published in 2005 by Bloomsbury. It is a humorous account of the effect of outsiders on the rural status quo in a fictional mid-Wales valley during the 1980s and is being adapted for television.", "category": "WrittenWork"}
|
||||
{"text": " Sema Group plc was an Anglo-French IT services company. It was listed on the London Stock Exchange and was a constituent of the FTSE 100 Index but was acquired by Schlumberger in 2001.", "category": "Company"}
|
||||
{"text": " Bent Hansen (born 1954) is a retired Danish ice hockey forward. He played for 18 years in Denmark for the R\u00f8dovre SIK and KSF. He also competed for the Danish national team. His son Jannik Hansen also played for the R\u00f8dovre team and was drafted into the NHL by the Vancouver Canucks in 2004. During his hockey career Hansen also worked as a carpenter.", "category": "Athlete"}
|
||||
{"text": " Behind the Sun is a 2004 album by Dive.", "category": "Album"}
|
||||
{"text": " Mungaru Male (English: Pre Monsoon Rain) is a 2006 Kannada language movie directed by Yograj Bhat and produced by E Krishnappa. The film stars Ganesh Pooja Gandhi Anant Nag Padmaja Rao in lead roles.", "category": "Film"}
|
||||
{"text": " Megachile perihirta commonly known as the Western leafcutting bee is a bee in the genus Megachile. The bee is native to western North America ranging from Nebraska to Texas and Mexico west to California and north to British Columbia and Alberta and often inhabits meadows and orchards. The bee is black with long whitish-yellow hair more so below the thorax and abdomen. The abdomen however is mostly bare although each segment has scattered whitish hair.", "category": "Animal"}
|
||||
{"text": " Sukeban Deka The Movie (\u30b9\u30b1\u30d0\u30f3\u5211\u4e8b) is a live action Japanese film that was released in 1987. The movie closely follows a TV and manga series Sukeban Deka written and illustrated by Shinji Wada. The movie stars Yoko Minamino and Yui Asaka who were also in the TV series. The movie was followed by Sukeban Deka II in 1988.", "category": "Film"}
|
||||
{"text": " The Maple School District is a public school district in Douglas County Wisconsin United States based in Maple Wisconsin.", "category": "EducationalInstitution"}
|
||||
{"text": " Mount Waverley Secondary College is a public secondary school located in the Melbourne suburb of Mount Waverley. The school consists of roughly 1900 students and is one of the largest in the state.The school consists of two campuses (Junior & Senior) both situated on Stephensons Road in Mount Waverley. The Junior site holds years 7 and 8 with year levels 9 to 12 at the Senior Campus. The campuses are a short walking distance apart.", "category": "EducationalInstitution"}
|
||||
{"text": " Jon-Paul Roger JP Pietersen (born 12 July 1986 in Stellenbosch South Africa) is a South African rugby union footballer. He generally plays fullback or wing for the Sharks (in the Super Rugby competition) and the Natal Sharks in the Currie Cup. He played in more than 50 tests for the Springboks.", "category": "Athlete"}
|
||||
{"text": " Deltocolpodes is a genus of beetles in the family Carabidae containing the following species: Deltocolpodes brendelli Morvan 1992 Deltocolpodes championi Morvan 1992 Deltocolpodes duluchus Morvan 1992 Deltocolpodes heinigeri Morvan 1992 Deltocolpodes jalepensis Morvan 1992 Deltocolpodes kirschenhoferi Morvan 1992 Deltocolpodes nepalensis Morvan 1992 Deltocolpodes perreaui Deuve 1985 Deltocolpodes rectangulus Morvan 1992 Deltocolpodes rolex Morvan 1992 Deltocolpodes salpensis Deuve 1985 Deltocolpodes sikkimensis Morvan 1992\u2191", "category": "Animal"}
|
||||
{"text": " Stanhopea martiana is a species of orchid endemic to southwestern Mexico.", "category": "Plant"}
|
||||
{"text": " Yawarmayu (Quechua yawar blood mayu river blood river hispanicized spelling Yahuarmayo) is a river in Peru located in the Puno Region Carabaya Province Ayapata District. It originates near the border of the districts Ayapata and Coasa. Its direction is mainly to the northwest where it meets Inambari River as a right affluent. The confluence is north of the village Yawarmayu (Yahuarmayo).", "category": "NaturalPlace"}
|
||||
{"text": " The Charles Granke House at 406 S. Seventh St. in Hamilton Montana is a historic house that was built in 1906. It includes Colonial Revival and Queen Anne architecture. It was listed on the National Register of Historic Places in 1988. The listing included two contributing buildings.It was built in approximately 1906 by the Anaconda Copper Mining Company as a worker cottage for workers at the sawmill that operated in Hamilton until 1915. Charles W.", "category": "Building"}
|
||||
{"text": " Passiflora monadelpha is a species of plant in the Passifloraceae family. It is endemic to Ecuador.", "category": "Plant"}
|
||||
{"text": " Mangifera persiciformis or Peach Mango is a species of plant in the Anacardiaceae family. It is endemic to China.", "category": "Plant"}
|
||||
{"prompt":" Morada Limited is a textile company based in Altham Lancashire. Morada specializes in curtains.","completion":"Company"}
|
||||
{"prompt":" The Armenian Mirror-Spectator is a newspaper published by the Baikar Association in Watertown Massachusetts.","completion":"WrittenWork"}
|
||||
{"prompt":" Mt. Kinka (\u91d1\u83ef\u5c71 Kinka-zan) also known as Kinkazan is located in the heart of the city of Gifu Gifu Prefecture Japan and rises to a height of 329 m (1079 ft). Previously called Mt. Inaba (\u7a32\u8449\u5c71 Inaba-yama) it has long served as the representative symbol of Gifu. It stands along the Nagara River creating bountiful nature within the city. Though it is the most famous mountain in the city Mount Dodo to the north is the tallest.","completion":"NaturalPlace"}
|
||||
{"prompt":" Planning the Play of a Bridge Hand is a book on contract bridge co-written by Canadian teacher and author Barbara Seagram and British author David Bird. It was published by Master Point Press in 2009.The book teaches novice bridge players some basic techniques of declarer play including suit establishment ruffing losers and the finesse.","completion":"WrittenWork"}
|
||||
{"prompt":" Wang Yuanping (born 8 December 1976) is a retired Chinese athlete who specialised in the 800 metres. She won several medals at the regional level.Her personal bests in the event are 2:00.63 seconds outdoors (Jinzhou 2000) and 2:03.41 seconds indoors (Yokohama 2004).","completion":"Athlete"}
|
||||
{"prompt":" The Incorporated VIllage of Westhampton Beach is an incorporated village in the Town of Southampton Suffolk County New York United States. As of the 2010 census the village population was 1721.","completion":"Village"}
|
||||
{"prompt":" Andersons Creek is a creek in Warrandyte and Park Orchards east of Melbourne Victoria Australia. It is a tributary of the Yarra River.","completion":"NaturalPlace"}
|
||||
{"prompt":" The Three Horseshoes is a public house in Drybridge Street in the Overmonnow area of Monmouth Wales. The pub has also been used as an Inn and also known as The Three Horse Shoes Inn. The building has been a Grade II Listed building since 15 August 1974. 19th century 2 storeys roughcast as stone with a hooded doorway","completion":"Building"}
|
||||
{"prompt":" The Brewer's Art is a Baltimore Maryland brewpub and restaurant. Opened on Friday September 13 1996. In 2008 it was named by Esquire magazine as the #1 Best Bar in America.","completion":"Company"}
|
||||
{"prompt":" The P\u00e2r\u00e2ul S\u0103r\u0103\u021bii is a tributary of the Cibin River in Romania.","completion":"NaturalPlace"}
|
||||
{"prompt":" Jean-Fran\u00e7ois Imbernon (born October 17 1951 in Perpignan France is a retired French international rugby union player.He played as a Lock for USA Perpignan. He earned his first cap with the French national team on 7 February 1976 against Ireland at Parc des Princes.","completion":"Athlete"}
|
||||
{"prompt":" Le Cadeau released in Italy as Il regalo is a 1982 French and Italian film. It stars Claudia Cardinale.","completion":"Film"}
|
||||
{"prompt":" Mykola Kanevets (Ukrainian: \u041c\u0438\u043a\u043e\u043b\u0430 \u041a\u0430\u043d\u0456\u0432\u0435\u0446\u044c) is the Artistic Director and Ballet Master of the Cheremosh Ukrainian Dance Company in Edmonton Alberta Canada.A native of Kiev Ukraine Mykola attended the National University of Culture and Performing Arts in Kiev Ukraine where he graduated from the Faculty of Choreography with the distinction of Ballet Master and Choreographer.","completion":"Artist"}
|
||||
{"prompt":" Jenna Rose Swerdlow (born September 28 1998) is an American teenage singer who gained media attention as a pre-teen with her single My Jeans. After the video went viral on YouTube and received 14 million views Swerdlow is considered a semi-viral star.","completion":"Artist"}
|
||||
{"prompt":" The Spice of Life is a smooth jazz studio album by Earl Klugh released in April 2008. The album received a Grammy nomination for Best Pop Instrumental Album at the 51st Grammy Awards in 2009.","completion":"Album"}
|
||||
{"prompt":" Lomatium macrocarpum is a perennial flowering plant in the carrot family known by the common names bigseed lomatium biscuit root or even bigseed biscuitroot. It is native to much of western North America where it can be found in various types of habitat including the grasslands of the Great Plains. It is spreading or erect perennial herb growing up to about half a meter long with hairy gray-green herbage.","completion":"Plant"}
|
||||
{"prompt":" Physena is the sole genus of the flowering plant family Physenaceae. It contains two species of shrubs and small trees which are endemic to Madagascar. The APG II system of 2003 (unchanged from the APG system of 1998) does recognize this family and assigns it to the order Caryophyllales in the clade core eudicots.","completion":"Plant"}
|
||||
{"prompt":" David John Weatherley (born 1 March 1939) is a New Zealander actor known for his roles as Spencer the butler and the voice of Benglo the Fearcat in Power Rangers Operation Overdrive and Barliman Butterbur in The Lord of the Rings: The Fellowship of the Ring.Weatherley was born in London England and moved to Canada for a military career. He eventually moved to New Zealand to engage in a theatre acting career.","completion":"Artist"}
|
||||
{"prompt":" Draba incrassata is an uncommon species of flowering plant in the mustard family known by the common name Sweetwater Mountains draba. It is endemic to California where it is known mainly from the Sweetwater Mountains of Mono County. It grows in alpine rock fields on the barren high mountain peaks. Draba incrassata is a small perennial herb forming mats of thick oval-shaped leaves.","completion":"Plant"}
|
||||
{"prompt":" Pimelea ferruginea is a small shrub native to southwest Western Australia. It was described by Labillardiere in 1805.","completion":"Plant"}
|
||||
{"prompt":" Lindsay Ell is a country music singer songwriter and guitarist from Calgary Alberta.She performed at the South by Southwest music festival held in Austin Texas in March 2009 the welcome reception of the 2009 Juno Awards held in Vancouver British Columbia and was also a featured artist at the 2010 Winter Olympics.","completion":"Artist"}
|
||||
{"prompt":" Scopula fuscata is a moth of the Geometridae family. It is found from south-western Saskatchewan west to British Columbia and south to California and Arizona. The habitat consists of montane areas including foothills.The wingspan is 24-28 mm. The wings and body are light tan sprinkled with darker yellow-brown or grey-brown scales. There is one generation per year with adults on wing in late June and early July in the northern part of the range.","completion":"Animal"}
|
||||
{"prompt":" Oxmoor Center is a Louisville Kentucky shopping mall located at 7900 Shelbyville Road in eastern Louisville.","completion":"Building"}
|
||||
{"prompt":" Ghostquake (also known as Haunted High) is a 2012 American made-for-television horror film produced by Syfy. The film was directed by Jeffrey Lando and written by Paul A. Birkett and Anthony C. Ferrante. The film stars Danny Trejo and MC Gainey. It follows a group of high school students trying to escape the wrath of a few ghastly spirits following an earthquake at their school Holloman High School.","completion":"Film"}
|
||||
{"prompt":" The Masonic Temple in Great Falls Montana is a building from 1914. It was listed on the National Register of Historic Places in 2000.Address is 821 Central Avenue Great Falls Motana 59401 Phone number is 453-9080.Cascade No. 34 meets 2nd and 4th Tuesdays at 7:30pm Sept-June.Euclid No. 58 meets year-round 1st and 3rd Tuesdays at 7:30pm Sept-May 3rd Tuesdays at 7:30pm June-Aug. Delta No. 128 meets 2nd Wednesdays at 7:30pm Sept-June.","completion":"Building"}
|
||||
{"prompt":" Harold Frederick Weaver Hawkins (1893-1977) was an English painter who specialized in ambitious sometimes mural-sized modernist allegories of morality for an age of atomic warfare and global over-population.","completion":"Artist"}
|
||||
{"prompt":" Robert Murray Waddington (24 October 1927 \u2013 15 March 2007) was Dean of Manchester in the last quarter of the 20th century.Born in Bognor Regis on 24 October 1927 he was educated at Dulwich College Selwyn College Cambridge and Ely Theological College. Ordained in 1954 he began his career at St John\u2019s Bethnal Green. Afterwards he was Chaplain at Slade School in Warwick Queensland. He returned to England in 1959 to join the Oratory of the Good Shepherd an order of celibate priests.","completion":"OfficeHolder"}
|
||||
{"prompt":" Jason Gary King (born 13 April 1985 in Maidstone England) is a speedway rider who was formerly the club captain of Newcastle Diamonds in the British Premier League. His brother Daniel is also a speedway rider.","completion":"Athlete"}
|
||||
{"prompt":" The African Queen is a 1951 adventure film adapted from the 1935 novel of the same name by C. S. Forester. The film was directed by John Huston and produced by Sam Spiegel and John Woolf. The screenplay was adapted by James Agee John Huston John Collier and Peter Viertel. It was photographed in Technicolor by Jack Cardiff and had a music score by Allan Gray.","completion":"Film"}
|
||||
{"prompt":" The Fiat Barchetta (Italian pronunciation: [\u02c8fiat bar\u02c8ketta]) (Type 183) is a roadster produced by the Italian manufacturer Fiat from 1995 to 2005 (though production was paused between May 2002 and 2004). Barchetta in Italian means 'little boat'.","completion":"MeanOfTransportation"}
|
||||
{"prompt":" Sardar Vallabhbhai Patel National Memorial is a museum and exhibition centre dedicated to Sardar Vallabhbhai Patel at Moti Shahi Mahal located in Shahibaug Ahmedabad Gujarat. Moti Shahi Mahal was constructed by Mughal emperor Shahjahan between 1618 and 1622. It is surrounded by a garden.","completion":"Building"}
|
||||
{"prompt":" Under Cover 2 is the 5th solo album of Joe Lynn Turner released in 1999. Just like Under Cover the album consists mainly of covers of Turner's favourite artists.","completion":"Album"}
|
||||
{"prompt":" The Atakora River is a tributary of Lake Volta in Ghana it flows about 60 km east to the Lake Volta. Its entire course is in south Ghana.","completion":"NaturalPlace"}
|
||||
{"prompt":" Death from Above is a 2011 horror film by director Bruce Koehler. The film features professional wrestling stars Kurt Angle Sid Eudy James Storm Matt Morgan Terry Gerin and Jessica Kresa.","completion":"Film"}
|
||||
{"prompt":" Portraits of Cuba is an album by Cuban musician Paquito D'Rivera released through Chesky Records in 1996. In 1997 the album won D'Rivera the Grammy Award for Best Latin Jazz Performance.","completion":"Album"}
|
||||
{"prompt":" Jimmy Cross (17 November 1938 - 8 October 1978) was an American radio producer and singer who attained a minor Billboard Hot 100 hit with the novelty song I Want My Baby Back in 1965. He was born in Dothan Alabama[citation needed] and became the producer of the syndicated radio series Country Concert.I Want My Baby Back was originally issued on the Tollie label and reached #92 on the Billboard Hot 100 in February 1965.","completion":"Artist"}
|
||||
{"prompt":" Timothy Floyd Tim Burchett (born August 25 1964) is an American Republican politician currently the mayor of Knox County Tennessee. He previously served in Tennessee General Assembly first in the Tennessee House of Representatives and later in the Tennessee State Senate in which he represented Tennessee's District 7 part of Knox County. On August 5 2010 Burchett was elected mayor of Knox County replacing Mike Ragsdale.","completion":"OfficeHolder"}
|
||||
{"prompt":" Daniel Lawrence Dan Whitney (born February 17 1963) best known by his stage name and character Larry the Cable Guy is an American stand-up comedian actor voice actor and former radio personality.","completion":"Artist"}
|
||||
{"prompt":" Renealmia is a plant genus in the family Zingiberaceae. Species include: Renealmia alpinia Renealmia aurantifera Renealmia cernua Renealmia dolichocalyx Renealmia oligotricha Renealmia sessilifolia Renealmia thrysoidesE.g. Alpinia nutans was formerly placed herein too.","completion":"Plant"}
|
||||
{"prompt":" Jeff Chapman (born July 17 1969 in Brunswick Georgia) is the bass singer for the Kingdom Heirs. He has been a member of the group since 2002. He has previously traveled with Bob Wills The Sound The Anchormen and The Blackwoods.He has twice been nominated as favorite bass in the Singing News fan awards.Chapman has a wife Angie two sons Justin and Sean and daughter Taylor.","completion":"Artist"}
|
||||
{"prompt":" Arenaria ursina is a species of flowering plant in the pink family known by the common name Bear Valley sandwort.","completion":"Plant"}
|
||||
{"prompt":" Living Fossil is a classic science fiction story on the concepts of human extinction and future evolution by L. Sprague de Camp. It was first published in the magazine Astounding Science-Fiction for February 1939. It first appeared in book form in the anthology A Treasury of Science Fiction (Crown Publishers 1948); it later appeared in the anthologies Gates to Tomorrow (Atheneum 1973) and The SFWA Grand Masters Volume 1 (Tor Books 1999).","completion":"WrittenWork"}
|
||||
{"prompt":" Brachyglottis huntii commonly called rautini or Chatham Island Christmas tree is a species in the Asteraceae family and is found only on the Chatham Islands in New Zealand.","completion":"Plant"}
|
||||
{"prompt":" Luktvatnet is a lake that lies in the northern part of the municipality of Vefsn in Nordland county Norway. The lake lies between the mountains Korgfjellet and Lukttinden about 5 kilometres (3.1 mi) south of Elsfjord. The European route E06 highway passes along the northern shore of the lake.","completion":"NaturalPlace"}
|
||||
{"prompt":" The IAR 79 is a bi-engine bomber military reconnaissance aircraft with a wood and metal structure that saw service in World War II built under licence in Brasov Romania by Industria Aeronautic\u0103 Rom\u00e2n\u0103","completion":"MeanOfTransportation"}
|
||||
{"prompt":" Enrico Perucconi (born 4 January 1925 in Morazzone Varese Italy) was an Italian athlete who competed mainly in the 100 metres.","completion":"Athlete"}
|
||||
{"prompt":" Central National-Gottesman Inc. is one of the world's largest distributors of pulp paper paperboard and newsprint. The firm's products are sold in over 75 countries through a network of 43 offices located in the United States and abroad. With annual revenues exceeding $3 billion Forbes ranked Central National-Gottesman 137th in its annual list of The Largest Private Companies.","completion":"Company"}
|
||||
{"prompt":" The Kout Food Group is a Kuwaiti-based conglomerate founded in 1982.In Kuwait it operates franchises of Burger King Pizza Hut and Taco Bell.Its UK arm Kout Food Group Restaurants UK it operates under brands such as Burger King KFC and Maison Blanc. In August 2013 it acquired the Little Chef chain for \u00a315 million.","completion":"Company"}
|
||||
{"prompt":" Fab Five: The Texas Cheerleader Scandal is a Lifetime Television made-for-TV drama film starring Jenna Dewan Ashley Benson and Tatum O'Neal and directed by Tom McLoughlin. The film premiered on August 2 2008. It is based on a true story which occurred at McKinney North High School in McKinney Texas in 2006 in which five teenage cheerleaders became notorious for bullying truancies violations of the school dress code and general disrespect to the school community and authority.","completion":"Film"}
|
||||
{"prompt":" Qadi Mahalleh (Persian: \u0642\u0627\u062f\u064a \u0645\u062d\u0644\u0647\u200e also Romanized as Q\u0101d\u012b Ma\u1e29alleh) is a village in Pazevar Rural District Rudbast District Babolsar County Mazandaran Province Iran. At the 2006 census its population was 228 in 59 families.","completion":"Village"}
|
||||
{"prompt":" Eungella Dam is one of Queensland's more established freshwater fisheries. Eungella has made a name for producing extra oversized Sooty grunter and more recently Barramundi.Eungella Dam was constructed in 1969 to meet the requirements of a thermal power station at Collinsville and the town water requirement of Collinsville and Scottsville.","completion":"NaturalPlace"}
|
||||
{"prompt":" The American Motor Car Company was a short-lived company in the automotive industry founded in 1906 lasting until 1913. It was based in Indianapolis Indiana United States. The American Motor Car Company pioneered the underslung design.","completion":"Company"}
|
||||
{"prompt":" Hawkeye & Mockingbird was a comic book ongoing series published by Marvel Comics starring superheroes Hawkeye and Mockingbird.","completion":"WrittenWork"}
|
||||
{"prompt":" Margaret Anderson Kelliher (born March 11 1968) is a Minnesota politician and a former member of the Minnesota House of Representatives. A member of the Minnesota Democratic\u2013Farmer\u2013Labor Party she represented District 60A which includes portions of the city of Minneapolis in Hennepin County located in the Twin Cities metropolitan area. First elected in 1999 she served until 2011 also serving as the Speaker from 2007 to 2011.","completion":"OfficeHolder"}
|
||||
{"prompt":" John Whitlow Wyatt (September 27 1907 \u2013 July 16 1999) was a professional baseball pitcher. He played all or part of sixteen seasons in Major League Baseball for the Detroit Tigers (1929\u201333) Chicago White Sox (1933\u201336) Cleveland Indians (1937) Brooklyn Dodgers (1939\u201344) and Philadelphia Phillies (1945). While injuries sidetracked much of Wyatt's early career he is most famous for his performance in 1941 when his team (the Dodgers) won the National League pennant.","completion":"Athlete"}
|
||||
{"prompt":" William Thomas Burton (31 January 1878 in Black Rock St Michael Barbados \u2013 22 August 1946 St Michael Barbados) was a coloured West Indian cricketer best known as a member of the 1900 and 1906 West Indian tourists to England. He is generally known as Tommie Burton.He was the son of a black mother and a white father. He was brought up in Barbados and served for some years there as a practice bowler and in trial matches.","completion":"Athlete"}
|
||||
{"prompt":" Tulemalu Lake is a lake in Kivalliq Region Nunavut Canada.","completion":"NaturalPlace"}
|
||||
{"prompt":" Sten Stjernqvist is a Swedish former footballer who played as a forward.","completion":"Athlete"}
|
||||
{"prompt":" David Parlett (born 1939) is a games scholar from South London who has studied both card games and board games. His published works include many popular books on games and the more academic volumes The Oxford Guide to Card Games and The Oxford History of Board Games both now out of print. Parlett also invented a number of board games the most successful of which is Hare and Tortoise (1974). The German edition was awarded Spiel des Jahres (Game of the Year) in 1979.","completion":"Artist"}
|
||||
{"prompt":" Karl Nabersberg (sometimes written as Carl Nabersberg) was a German youth leader.Nabersberg was the son of a Crefeld shopkeeper. In 1923 he joined the Jugendorganisation the forerunner of the Hitler Youth in his home town. On 28 December 1925 he was admitted as a member of the National Socialist German Workers' Party (member number 26269) and as a member of the Sturmabteilung.","completion":"OfficeHolder"}
|
||||
{"prompt":" \u0160etonje is a village situated in Petrovac na Mlavi municipality in Serbia.","completion":"Village"}
|
||||
{"prompt":" Dr. Joseph de Graft-Johnson (1933\u20131999) was an engineer academic and politician. He became the Vice-President of Ghana between 1979 and 1981.","completion":"OfficeHolder"}
|
||||
{"prompt":" Patties Foods (previously Patties Bakery) is an Australian food manufacturing company that produces meat pies baked goods frozen fruits and pre-made desserts. Headquartered in Bairnsdale Victoria Australia Patties Foods is represented in the Australian market by the Four'N Twenty Patties Herbert Adams Creative Gourmet Nanna's and Chefs Pride brands. Patties is the largest meat pie producing company in Australia and the world.","completion":"Company"}
|
||||
{"prompt":" Double Butte is the 2579-foot (786 m) mountain summit distinguished by two buttes (the other at abou 2480 feet or 756 metres) in Riverside County California. It is the western most summit of a mountain range north of Winchester California east of Perris Valley and west of the San Jacinto Valley. The eastern ridge is composed primarily of metamorphic rock of the Triassic - Jurassic French Valley formation.","completion":"NaturalPlace"}
|
||||
{"prompt":" Mount Carmel \u2013 Blytheswood Public School is an elementary school in the north end of Leamington Ontario Canada. It is part of the Greater Essex County District School Board and serves students from JK to Grade 8 from the communities of Blytheswood and Mount Carmel and surrounding areas.","completion":"EducationalInstitution"}
|
||||
{"prompt":" La combi asesina (The Killer Combination) is a 1982 Mexican film. It was directed by Gustavo Alatriste.","completion":"Film"}
|
||||
{"prompt":" Halimium ocymoides (basil-leaved rock rose) syn. Cistus algarvensis is a species of flowering plant in the family Cistaceae native to southern Portugal and southern Spain. It is an erect evergreen shrub growing to 60 cm (24 in) tall by 100 cm (3 ft) wide with woolly grey-green leaves and bright yellow flowers in spring. The flowers may have a dark brown blotch at the base of each petal.In cultivation this plant requires a sandy soil and full sun.","completion":"Plant"}
|
||||
{"prompt":" Kaala Patthar (English: Black Stone) is a 1979 Indian Bollywood action\/drama film. It was produced and directed by Yash Chopra. The story was written by Salim-Javed. This film is the fourth collaboration between Amitabh Bachchan Shashi Kapoor and director Yash Chopra after the hugely successful Deewaar (1975) Kabhie Kabhie (1976) and Trishul (1978). However this film did average business at the box office. It was nominated for Filmfare awards.","completion":"Film"}
|
||||
{"prompt":" Martin G.S. Mansergh (born 31 December 1946) is a former Irish Fianna F\u00e1il politician and historian. He was a Teachta D\u00e1la (TD) for the Tipperary South constituency from 2007 until 2011. He was previously a Senator from 2002 to 2007. He played a leading role in formulating Fianna F\u00e1il policy on Northern Ireland.","completion":"OfficeHolder"}
|
||||
{"prompt":" Shriniwas Ganesh Sardesai (1907-1996) popularly known as S.G. Sardesai was an Indian freedom fighter from Maharashtra and one of the great communist leaders produced by the communist movement in India. He is author of the book Progress and conservatism in ancient India famous for his profound theoretical analysis. He was the Central Executive Committee of pre-split Communist Party of India during the Indo-china conflict.","completion":"OfficeHolder"}
|
||||
{"prompt":" USS Tuluran (AG-46) \u2013 also known as USS Lake Superior (ID-2995) \u2013 was a commercial cargo ship acquired by the U.S. Navy for service during both World War I when she was known as USS Lake Superior and also during World War II when she was known as USS Tuluran.","completion":"MeanOfTransportation"}
|
||||
{"prompt":" The American Journal of Gastroenterology is a peer-reviewed medical journal published for the American College of Gastroenterology by the Nature Publishing Group.","completion":"WrittenWork"}
|
||||
{"prompt":" William Lindsay (September 4 1835 \u2013 October 15 1909) was a Democratic U.S. Senator from Kentucky from 1893 to 1901.Born near Lexington Virginia Lindsay attended the common schools and settled in Clinton Kentucky in 1854. There he taught school and studied law. He was admitted to the bar and commenced practice in Clinton in 1858.","completion":"OfficeHolder"}
|
||||
{"prompt":" Brian Schroeder (a.k.a. Pushead) is an artist record label owner and writer within the hardcore punk and heavy metal field. He has created artwork for many bands artists and athletes including Metallica The Misfits Dr. Dre Travis Barker Craig Johnson and Kool Keith. He has designed many record covers T-shirts skateboards and a pair of Nike SB Dunks. His record label Pusmort Records has released albums by Negative Gain Poison Idea and Final Conflict.","completion":"Artist"}
|
||||
{"prompt":" Panicum anceps is a species of grass known by the common name beaked panicgrass. It is native to the southeastern United States where it occurs as far north as New Jersey and as far west as Kansas and Texas.This species is a rhizomatous perennial grass with stems growing up to 1.3 meters tall. The leaves have erect blades up to half a meter tall. The inflorescence is a panicle up to 40 centimeters long bearing pale green or yellowish spikelets. The grass produces an abundance of seed.","completion":"Plant"}
|
||||
{"prompt":" Shukan ST is a weekly newspaper published by The Japan Times for learners of English language. It is originally titled as Student Times but changed to Shukan ST since a significant portion of its readers are not students. It has articles on news movie lifestyle in English-speaking countries opinions and other kinds attracting learners of English and helping them with notes on terms.","completion":"Company"}
|
||||
{"prompt":" The Tiger Hotel is a hotel in Columbia Missouri. Built as a hotel in 1928 the building later housed a retirement home and banquet center. In 2012 the building was fully restored and reopened as a boutique hotel. It was listed on the National Register of Historic Places in 1980.","completion":"Building"}
|
||||
{"prompt":" Emi Motoi (\u672c\u4e95 \u3048\u307f Motoi Emi born October 11 in Kanagawa) is a Japanese voice actress.","completion":"Artist"}
|
||||
{"prompt":" The Hudson River is a 49.5-mile-long (79.7 km) tributary of the Broad River in the U.S. state of Georgia. Via the Broad River it is part of the Savannah River watershed.The headwaters are in Banks County near the city of Homer. Grove Creek feeds into the Hudson near the Franklin County line. The river then constitutes most of the southern border of Franklin County separating it from Madison County.","completion":"NaturalPlace"}
|
||||
{"prompt":" This article details Car Nos. 10\u201313 of the Manx Electric Railway on the Isle of Man.This was the third batch of motorcars delivered to the railway in 1895 at the same time as the cars for the new Snaefell Mountain Railway were delivered. They were constructed to a very similar design to those provided for the mountain line.","completion":"MeanOfTransportation"}
|
||||
{"prompt":" Catharanthus roseus commonly known as the Madagascar rosy periwinkle is a species of Catharanthus native and endemic to Madagascar. Other English names occasionally used include Cape periwinkle rose periwinkle rosy periwinkle and old-maid.","completion":"Plant"}
|
||||
{"prompt":" Thapanzeik is a village in Homalin Township Hkamti District in the Sagaing Region of northwestern Burma.","completion":"Village"}
|
||||
{"prompt":" USS Spiegel Grove (LSD-32) was a Thomaston-class dock landing ship of the United States Navy. She was named for Spiegel Grove the home and estate in Fremont Ohio of Rutherford B. Hayes the 19th President of the United States.","completion":"MeanOfTransportation"}
|
||||
{"prompt":" Acmella is a genus of thirty species of plants in the aster family Asteraceae. It is native to the Americas and has been introduced to Asia Africa the Pacific islands and Australia.One familiar species is Acmella oleracea which has been widely cultivated for centuries. It is used for food and medicine and as an insecticide and an ornamental plant.","completion":"Plant"}
|
||||
{"prompt":" Mirbelia is a plant genus belonging to the Fabaceae family. It is endemic to Australia occurring in every mainland state except South Australia.","completion":"Plant"}
|
||||
{"prompt":" Nigma puella is a species of spider belonging to the family Dictynidae. It is found in Europe Azores Madeira Canary Islands and parts of North Africa.Like most members of the family this is a small spider but the female is striking with a light green abdomen marked with a bold maroon blotch and a variable amount of barring in the same colour. The male is reddish-brown. This species makes a horizontal web over the top surface of a leaf.","completion":"Animal"}
|
||||
{"prompt":" The Madrisa (or Madrisahorn) is a mountain in the R\u00e4tikon mountain range overlooking Klosters in the Swiss canton of Graub\u00fcnden. Its summit (2826 metres) is located near the Austrian border.The Madrisa is constituted by several secondary summits notably the Gargeller Madrisa (2770 metres) overlooking Gargellen in Austria.Ski lifts up to 2600 metres are located on the Klosters side.","completion":"NaturalPlace"}
|
||||
{"prompt":" Temporary Temple is a live album by Psychic TV. The album was recorded on July 28 1984 in London and released on 12 vinyl. It was later coupled with another concert and released on CD as Temporary Temple & Atonal.","completion":"Album"}
|
||||
{"prompt":" La hija de Juan Sim\u00f3n (Juan Sim\u00f3n's Daughter) is a musical play by Nemesio M. Sobrevila which has been made into two Spanish films. It is also the name of the title track and the song has been recorded by numerous artists such as Leonardo Favio.The first film directed by Jos\u00e9 Luis S\u00e1enz de Heredia was released in 1935 and starred Angelillo Pilar Mu\u00f1oz and Manuel Arb\u00f3. Luis Bu\u00f1uel was the executive producer for Film\u00f3fono and had a small role as an actor.","completion":"Film"}
|
||||
{"prompt":" Book Of Matches is a poetry book written by Simon Armitage first published in 1993 by Faber and Faber. Several poems featured in the book are studied as part of the GCSE English Literature examination in the UK.The book is written in three sections the first (Book of Matches) containing 30 short sonnets. Each is meant to be read within 20 seconds the amount of time it would take for a match to be lit and burn out.","completion":"WrittenWork"}
|
||||
{"prompt":" The Last Supper is the fourth album released by American stand-up comedian Jim Gaffigan. It focuses largely on his love of food.","completion":"Album"}
|
||||
{"prompt":" The Miami Center is a skyscraper in downtown Miami Florida. Although Miami Center is not the city's tallest building it is a symbol of early downtown. Built in 1983 it is older compared with most of the taller buildings in Miami which have been built in the last decade. In addition the Miami Center is immediately adjacent to Bayfront Park and is unobstructed when looking at the skyline from Miami Beach to the east. The building is 484 ft (148 m) tall and has 34 floors.","completion":"Building"}
|
||||
{"prompt":" Duboisia hopwoodii is a shrub native to the arid interior region of Australia. Common names include pituri pitchuri thornapple or pitcheri. It has an erect habit usually growing to between 1 and 3 metres in height and has long narrow leaves. Flowers are white and bell-shaped with violet-striped throats. These appear between June and November in the species native range followed by purple-black rounded berries which are 3 to 6 mm in diameter.","completion":"Plant"}
|
||||
{"prompt":" Jelenin svet (Jelena's World) is a 2008 independent documentary film written and directed by Tanja Brzakovi\u0107 about former World No. 1 female tennis player Jelena Jankovi\u0107.","completion":"Film"}
|
||||
{"prompt":" Jay Cashman Inc. is an American heavy-construction company based in Quincy Massachusetts with satellite offices in Boston Jupiter Florida and Staten Island New York. As of 2006 the company has about 1000 employees. The company was one of the major contractors on the Boston's Central Artery\/Tunnel Project. In 2004 Jay Cashman Inc.","completion":"Company"}
|
||||
{"prompt":" Hashemanli (Persian: \u0647\u0627\u0634\u0645\u0646\u0644\u064a\u200e also Romanized as H\u0101shemanl\u012b; also known as H\u0101shem El\u00e1) is a village in Jafarbay-ye Jonubi Rural District in the Central District of Torkaman County Golestan Province Iran. At the 2006 census its population was 135 in 27 families.","completion":"Village"}
|
||||
{"prompt":" Rani Kasula Rangamma is a Telugu film starring Chiranjeevi.","completion":"Film"}
|
||||
{"prompt":" The 20\/20 Experience \u2013 The Complete Experience is a compilation album by American singer-songwriter Justin Timberlake. It was released on September 27 2013 by RCA Records.","completion":"Album"}
|
||||
{"prompt":" R.C. Bigelow Inc better known as the Bigelow Tea Company is an American tea company based in Fairfield Connecticut. The company was founded by Ruth C. Bigelow in the late 1940s based on a recipe she marketed as Constant Comment tea. Bigelow is still a 100% family-owned business that markets over 50 varieties of tea including black and green as well as herbal teas all of which are still blended in Fairfield. They also own America's only tea plantation in Charleston South Carolina.","completion":"Company"}
|
||||
{"prompt":" Thomas Eyre(fl. 1890s) was a footballer who made 65 appearances in the Football League playing for Lincoln City. He played at left back. Either side of Lincoln he played for Ashfield and Hamilton Academical in Scotland.","completion":"Athlete"}
|
||||
{"prompt":" Malleable Iron Range Company was a company that existed from 1896 to 1985 and primarily produced kitchen ranges made of malleable iron but also produced a variety of other related products. The company's primary trademark was 'Monarch' and was colloquially often referred to as the Monarch Company or just Monarch.","completion":"Company"}
|
||||
{"prompt":" The Chiltern School is a coeducational special school located over two sites in Dunstable and Houghton Regis in Bedfordshire England. The school accepts pupils from all over the Central Bedfordshire area.The school was formed in 2012 from the merger of Glenwood School in Dunstable and Hillcrest School in Houghton Regis.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Kim Dae-Eun (born September 17 1984) is a South Korean gymnast. He is the 2004 Olympic All-around silver medalist. He won the gold medal on the parallel bars at the 2007 World Artistic Gymnastics Championships.Kim was part of the South Korean team that won the bronze medal in the team event at the 2006 Asian Games.","completion":"Athlete"}
|
||||
{"prompt":" Arayik Vladimirovich Harutyunyan (Armenian: \u0531\u0580\u0561\u0575\u056b\u056f \u0540\u0561\u0580\u0578\u0582\u0569\u0575\u0578\u0582\u0576\u0575\u0561\u0576 Russian: \u0410\u0440\u0430\u0438\u043a \u0410\u0440\u0443\u0442\u044e\u043d\u044f\u043d) (born 14 December 1973) is the current Prime Minister of the Nagorno-Karabakh Republic. He was suggested by the President of Nagorno-Karabakh Bako Sahakyan and was unanimously approved by the Parliament of Karabakh on 14 September 2007 by 32 out of 32 present parliamentarians.","completion":"OfficeHolder"}
|
||||
{"prompt":" Shelton Hank Williams also known as Hank Williams III and Hank 3 (born December 12 1972) is an American musician singer and multi-instrumentalist including guitar bass drums banjo and vocals. In addition to his honky tonk recordings Williams' style alternates between country punk and metal.","completion":"Artist"}
|
||||
{"prompt":" Helicella orzai is a species of air-breathing land snails terrestrial pulmonate gastropod mollusks in the family Hygromiidae the hairy snails and their allies. This species is endemic to Spain.","completion":"Animal"}
|
||||
{"prompt":" Gro\u00dfe Schmalenau is a river of North Rhine-Westphalia Germany.","completion":"NaturalPlace"}
|
||||
{"prompt":" The Tupolev ANT-29 (military designation DIP \u2013 Dvukhmotorny istrebitel pushechny twin-engined cannon fighter) was a 1930s twin-engined cannon-armed fighter designed by Alexander Arkhangelsky and built by Tupolev.Design work started in 1932 on a twin-engined aircraft capable of carrying two APK-100 cannons. The resulting design was the ANT-29 and it first flew in February 1935. A monoplane with a tall and narrow fuselage and powered by two Hispano-Suiza 12Ybrs engines.","completion":"MeanOfTransportation"}
|
||||
{"prompt":" Charles Corm (1894-1963) was a Lebanese writer businessman and philanthropist. He is considered to be the leader of the Phoenicianism movement in Lebanon which ignited a surge of nationalism that led to Lebanon's independence. In a country torn by sectarian conflicts Corm's intention was to find a common root shared by all Lebanese beyond their religious beliefs (the Phoenicians were pagans).","completion":"Artist"}
|
||||
{"prompt":" Joseph Hubert Ruetz (October 21 1916 \u2013 January 2 2003) was a professional football player in the All-America Football Conference for the Chicago Rockets in 1946 and 1948. Prior to that he played at the collegiate level while attending the University of Notre Dame. He played guard for the Irish with the exception of playing one season at quarterback. In 1938 he graduated from Notre Dame with cum laude honors.","completion":"Athlete"}
|
||||
{"prompt":" The Reef House is a historic house located at 411 S. Poplar St. in Carbondale Illinois. William A. Reef built the house for his family circa 1892. The Queen Anne-style cottage may have been designed by local carpenter A. M. Etherton though records of its designer do not exist. The house features fishscale shingle siding on its second floor and clapboard siding on its first; the clapboard siding is adorned with stickwork.","completion":"Building"}
|
||||
{"prompt":" MAKO Surgical Corp. (Stryker Medical) is a publicly traded medical device company based in Florida. On September 25 2013 the Board of Directors of Mako Surgical accepted a deal to merge with Stryker Medical for $1.65B subject to shareholder approval.","completion":"Company"}
|
||||
{"prompt":" Pop Carn is a 2003 Indian Tamil film written and directed by actor-cum-director Nassar and starring Mohanlal and Simran Bagga in lead roles and introducing newcomers Kunal Shah and Jyothi Nawal. The film which had music scored by Yuvan Shankar Raja was released on 30 January 2003 but flopped at the box office. Nonetheless the film was dubbed into Malayalam and released in 2007 under the same name.","completion":"Film"}
|
||||
{"prompt":" USNS Mount Baker (T-AE-34) is the seventh of eight Kilauea-class ammunition ships to serve with the Military Sealift Command. She is the second U.S. Navy ship to bear the name and is named for Mount Baker a 10781-foot volcano in the Cascade Range of Washington. Ammunition ships operated by Military Sealift Command provide logistic support to US Navy ships at sea.Mount Baker was built by Ingalls Shipbuilding Pascagoula Mississippi.","completion":"MeanOfTransportation"}
|
||||
{"prompt":" Dansere is an album by Jan Garbarek. The album was recorded in November 1975 and features the Bobo Stenson Quartet.","completion":"Album"}
|
||||
{"prompt":" Divraz (Persian: \u062f\u064a\u0648\u0631\u0632\u200e also Romanized as D\u012bvraz) is a village in Bala Khiyaban-e Litkuh Rural District in the Central District of Amol County Mazandaran Province Iran. At the 2006 census its population was 393 in 95 families.","completion":"Village"}
|
||||
{"prompt":" The D\u0103ih\u0103\u021ba\u0219u River is a tributary of the Dumbr\u0103vanu River in Romania.","completion":"NaturalPlace"}
|
||||
{"prompt":" Zeisters also known as Fat Guy Goes Nutzoid is a 1986 comedy film produced by Troma Entertainment. Troma was originally set to title the film Fat Boy Goes Nutzoid but at the request of the lawyers of the hip-hop group The Fat Boys it was changed to Fat Guy.","completion":"Film"}
|
||||
{"prompt":" Paul Gobeil (born March 1 1942 in Saint-R\u00e9mi-de-Tingwick Quebec) is a former Canadian politician and businessman.From 1985 to 1989 Mr. Gobeil was a Liberal member of the National Assembly for the riding of Verdun and served as Minister assigned to Administration President of the Treasury Board and as Minister of International Affairs for the Government of Quebec.","completion":"OfficeHolder"}
|
||||
{"prompt":" Ruff Ryders: Past Present Future is the fifth compilation album from American hip hop record label Ruff Ryders Entertainment released on November 21 2011.","completion":"Album"}
|
||||
{"prompt":" Ridi Viharaya (Sinhala: \u0dbb\u0dd2\u0daf\u0dd3 \u0dc0\u0dd2\u0dc4\u0dcf\u0dbb\u0dba) or Silver Temple is a 2nd-century BCE Theravada Buddhist temple in the village of Ridigama Sri Lanka. Built during the reign of Dutthagamani of Anuradhapura the temple is considered as the place where the silver ore which provided silver to complete Ruwanwelisaya; one of the largest stupa in Sri Lanka was discovered.","completion":"Building"}
|
||||
{"prompt":" Grand Canyon Preparatory Academy is a public charter college preparatory school in Tempe Arizona.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Aricoceras is an extinct genus of the Adrianitidae family. They are an extinct group of ammonoid which are shelled cephalopods related to squids belemnites octopuses and cuttlefish and more distantly to the nautiloids.","completion":"Animal"}
|
||||
{"prompt":" Blackburn High School is a public secondary school for girls and boys in years 7 to 12 in Blackburn a suburb of Melbourne Victoria Australia. Blackburn High School is an outstanding secondary school for aspiring young men and women. It aims to educate tomorrow's minds today. Started in 1956 the school has a proud tradition of academic excellence and exceptional music achievement.The school is nationally recognised as a leading educational institution for music education.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Chris Nieratko (born February 19 1976) is an American humorist and author. Nieratko is a past editor of Big Brother Magazine and currently reviews pornographic films for Vice magazine as well as being the author of the related Skinema book. He also appeared on MTV's Jackass.","completion":"Artist"}
|
||||
{"prompt":" Warlock is a 1959 film released by Twentieth Century Fox and shot in DeLuxe Color and CinemaScope. It is a Western adapted from the novel by Oakley Hall (screenplay written by Robert Alan Aurthur).","completion":"Film"}
|
||||
{"prompt":" Sieniawa [\u0255e\u02c8\u0272ava] is a village in the administrative district of Gmina Raba Wy\u017cna within Nowy Targ County Lesser Poland Voivodeship in southern Poland. It lies approximately 5 kilometres (3 mi) south-east of Raba Wy\u017cna 10 km (6 mi) north-west of Nowy Targ and 59 km (37 mi) south of the regional capital Krak\u00f3w.The village has a population of 1900.","completion":"Village"}
|
||||
{"prompt":" Michael Adam (born 9 December 1984) is a German politician. He has been District Administrator (Landrat) of Regen since 2011.","completion":"OfficeHolder"}
|
||||
{"prompt":" Thunderbird High School is a public high school located in northwestern Phoenix Arizona. The school is a part of the Glendale Union High School District.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Nayef Al Khater (born May 10 1978) is a Qatari football player. He currently plays for Al Wakrah as a defender.","completion":"Athlete"}
|
||||
{"prompt":" Black Cobra Woman (Italian: Eva nera) also known as Black Cobra is an Italian 1976 exploitation movie written and directed by Joe D'Amato.","completion":"Film"}
|
||||
{"prompt":" Joe Cuba a.k.a Sonny (April 22 1931 \u2013 February 15 2009) was a musician of Puerto Rican descent who was known as the Father of Latin Boogaloo.","completion":"Artist"}
|
||||
{"prompt":" Jacob LeBlanc (born February 2 1981 in Auburn California) is an American retired professional soccer player.","completion":"Athlete"}
|
||||
{"prompt":" Kevin B. Kamenetz is the 12th and current County Executive of Baltimore County Maryland serving since 2010. He is a member of the Democratic Party. He previously served as a four-term County Councilman representing the Second District of Baltimore County.","completion":"OfficeHolder"}
|
||||
{"prompt":" Thomas Frederick Fred Peart Baron Peart PC (30 April 1914 \u2013 26 August 1988) was a British Labour politician who served in the Labour governments of the 1960s and 1970s and was a candidate for Deputy Leader of the Party.","completion":"OfficeHolder"}
|
||||
{"prompt":" Grand Lake is a large lake in the interior of Newfoundland of the Canadian province of Newfoundland and Labrador. It has an area of 534 km\u00b2 making it the largest lake on Newfoundland. Consequently it is one of if not the deepest.","completion":"NaturalPlace"}
|
||||
{"prompt":" A Colossal Failure of Common Sense: The Inside Story of the Collapse of Lehman Brothers is a 2009 non-fiction book written by Lawrence G. McDonald and Patrick Robinson which chronicles the events surrounding the bankruptcy of Lehman Brothers in the context of the financial crisis of 2007\u20132010 and the subprime mortgage crisis.","completion":"WrittenWork"}
|
||||
{"prompt":" Thatching (31 May 1975 \u2013 1999) was an Irish Thoroughbred racehorse and sire. The horse's early career was delayed and disrupted by injury and he did not show his best form until switched to sprinting distances in the spring of 1979 when he won the Duke of York Stakes. He improved further when equipped with blinkers that summer recording impressive victories in both the Cork and Orrery Stakes and the July Cup.","completion":"Animal"}
|
||||
{"prompt":" Ya\u015far Kurt (Armenian: \u0545\u0561\u0577\u0561\u0580 \u053f\u0578\u0582\u0580\u0569 b.August 16 1968 in Istanbul Turkey) is a Turkish-Armenian rock artist.","completion":"Artist"}
|
||||
{"prompt":" Then and Now is a historical novel by W. Somerset Maugham. Set in Florence Italy during the Renaissance the story focuses on three months in the life of Niccolo Machiavelli the Florentine politician diplomat philosopher and writer in the early years of the 16th century. The book was first published by Heinemann in 1946.","completion":"WrittenWork"}
|
||||
{"prompt":" Abdollah Masud-e Sofla (Persian: \u0639\u0628\u062f\u0627\u0644\u0647 \u0645\u0633\u0639\u0648\u062f\u0633\u0641\u0644\u064a\u200e also Romanized as \u2018Abdoll\u0101h Mas\u2018\u016bd-e Sofl\u00e1; also known as Abdollah Mas\u2019ood \u2018Abdoll\u0101h Mas\u2018\u016bd and Abdull\u0101h Mas\u016bd) is a village in Hesar-e Valiyeasr Rural District Avaj District Buin Zahra County Qazvin Province Iran. At the 2006 census its population was 72 in 15 families.","completion":"Village"}
|
||||
{"prompt":" Springhill High School (SHS) is a secondary school in Springhill Nova Scotia Canada. SHS is part of the Chignecto-Central Regional School Board and is the only high school in the town of Springhill. The school is home to many sports teams and clubs. These include: basketball soccer badminton track and field softball Students Against Destructive Decisions drama club homework club and book club.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Charniele L. Herring (\/\u0283\u0251r\u02c8n\u025bl \u02c8h\u025br\u026a\u014b\/ shar-NEL HERR-ing; born September 25 1969) is an American politician. She has served in the Virginia House of Delegates since 2009 representing the 46th district made up the city of Alexandria and part of Fairfax County near Washington D.C. Herring is a member of the Democratic Party. She has been the House minority whip since 2012 and in December 2012 she was the first African-American to be elected chair of the Democratic Party of Virginia.","completion":"OfficeHolder"}
|
||||
{"prompt":" Symmoca dodecatella is a moth of the Symmocidae family. It is found in Portugal and Spain.The wingspan is about 18\u201319 mm. The forewings are grey sprinkled with black mainly along the margin. The hindwings are grey.","completion":"Animal"}
|
||||
{"prompt":" Ali Abbasov Mammad oglu (Azerbaijani: \u018fli Abbasov M\u0259mm\u0259d o\u011flu) (born 1953 in Azerbaijan) is the current Minister of Communications and Information Technologies of the Republic of Azerbaijan.","completion":"OfficeHolder"}
|
||||
{"prompt":" Worlds Beyond was an American digest magazine of science fiction and fantasy fiction in 1950 and 1951. The magazine only issued three monthly issues from December 1950 to February 1951 but is notable for having printed stories by Cyril M. Kornbluth Jack Vance Mack Reynolds Graham Greene John Christopher Lester del Rey Judith Merril and others.Worlds Beyond was published by Hillman Periodicals and was edited by Damon Knight.","completion":"Company"}
|
||||
{"prompt":" The Daily News Journal commonly abbreviated to DNJ is a newspaper serving Murfreesboro Tennessee Rutherford County and surrounding communities. Published in Murfreesboro it serves as the primary local newspaper with competition from The Murfreesboro Post and other publications. The newspaper is not in competition with The Tennessean of Nashville as both are owned by Gannett Company.The roots the DNJ date back to 1849 and the founding of Murfreesboro News.","completion":"WrittenWork"}
|
||||
{"prompt":" Echinocereus fendleri is a species of cactus known by the common names pinkflower hedgehog cactus and Fendler's hedgehog cactus. It grows in deserts and woodlands in the Southwestern United States and Northeastern Mexico. It is most common in New Mexico.The taxonomy of the species is uncertain with authors recognizing up to eight varieties.","completion":"Plant"}
|
||||
{"prompt":" Michael F. Kitt Snr (13 September 1914 \u2013 24 December 1974) was an Irish Fianna F\u00e1il politician and long-serving Teachta D\u00e1la (TD).He was elected to D\u00e1il \u00c9ireann for the first time at the 1948 general election for the Galway North constituency but lost his seat at the 1951 general election and failed to be elected again at the 1954 general election.","completion":"OfficeHolder"}
|
||||
{"prompt":" Epidendrum mancum is an epiphytic orchid that grows in the tropical low elfin cloud forests of Ecuador and Amazonas Peru at altitudes of 2\u20143 km .","completion":"Plant"}
|
||||
{"prompt":" Salempur Masanda is a village in Jalandhar District near the Jalandhar Cantonment in Punjab India.","completion":"Village"}
|
||||
{"prompt":" Yaleh Gonbad (Persian: \u064a\u0644\u0647 \u06af\u0646\u0628\u062f\u200e; also known as Em\u0101mz\u0101deh Imamzade-Ele-Geumbez and Im\u0101mz\u0101deh) is a village in Ilat-e Qaqazan-e Gharbi Rural District Kuhin District Qazvin County Qazvin Province Iran. At the 2006 census its population was 429 in 105 families.","completion":"Village"}
|
||||
{"prompt":" Popeyes Louisiana Kitchen is an American chain of fried chicken fast food restaurants founded in 1972 in New Orleans Louisiana. Often referred to as Popeyes and sometimes as Popeyes Chicken & Biscuits or Popeyes Chicken & Seafood[citation needed] it was acquired by Sandy Springs Georgia-based AFC Enterprises originally America's Favorite Chicken Company in 1993.","completion":"Company"}
|
||||
{"prompt":" The White Umfolozi River originates just south of Vryheid KwaZulu-Natal South Africa and joins the Black Umfolozi River at 28\u00b020\u203258\u2033S 31\u00b058\u203246\u2033E to form the Umfolozi River before it flows east towards the Indian Ocean.","completion":"NaturalPlace"}
|
||||
{"prompt":" The Albatros L 74 was a two-seated German training biplane produced by Albatros Flugzeugwerke. Only two were produced.","completion":"MeanOfTransportation"}
|
||||
{"prompt":" The University of Nevada School of Medicine is an academic division of the University of Nevada Reno and grants the Doctor of Medicine (MD) degree. The School of Medicine was founded in 1969 as the first medical school in the state of Nevada. More than 1500 MDs have graduated from the School of Medicine. The pre-clinical campus is located in Reno but the third and fourth years can be spent in hospitals and clinics throughout Nevada.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Leon Kroll (December 6 1884 \u2013 October 25 1974) was an American painter and lithographer. Known as a figurative artist Life Magazine described him as the dean of U.S. nude painters yet he was an exceptional landscape painter and also produced an exceptional body of still life compositions.Born into a musical family on lower Second Avenue in New York City Kroll's father was a violinist and his cousin was William Kroll.","completion":"Artist"}
|
||||
{"prompt":" Michael E. DeBakey High School for Health Professions at Qatar (DHSHP@Q in short) is a private international middle and secondary school in Doha Qatar. The school is a branch campus of Michael E. DeBakey High School for Health Professions of Houston Texas United States. Charlesetta Deason is the CEO and President.Named after Michael E. DeBakey the school opened in September 2008 with grades 8 through 10 with 100 students per grade; the school will ultimately cover grades 7-12.","completion":"EducationalInstitution"}
|
||||
{"prompt":" The Richleighs of Tantamount is a children\u2019s historical novel written by British author Barbara Willard. It was originally published in the United Kingdom in 1966 by the publishers Constable before being published in the United States by Harcourt Brace & World in June 1967. C. Walter Hodges drew the line illustrations and painted the cover portrait for the original edition.","completion":"WrittenWork"}
|
||||
{"prompt":" Ennea is a genus of air-breathing land snails terrestrial pulmonate gastropod mollusks in the family Streptaxidae.Ennea is the type genus of the subfamily Enneinae.","completion":"Animal"}
|
||||
{"prompt":" Come Live with Me is a 1941 American romantic comedy film produced and directed by Clarence Brown and starring James Stewart and Hedy Lamarr. Based on a story by Virginia Van Upp the film is about a beautiful Viennese refugee seeking United States citizenship who arranges a marriage of convenience with a struggling writer.","completion":"Film"}
|
||||
{"prompt":" St. Thomas Episcopal Church is a parish church in the Episcopal Diocese of Iowa. The church is located in Sioux City Iowa United States at 1200 Douglas Street. The church building is listed on the National Register of Historic Places.","completion":"Building"}
|
||||
{"prompt":" Nuno Daniel Costeira Valente (born 22 November 1991 in Ada\u00fafe - Braga ) is a Portuguese footballer who plays for Vizela on loan from S.C. Braga as a midfielder.","completion":"Athlete"}
|
||||
{"prompt":" Jaagoo (Bengali: \u099c\u09be\u0997\u09cb) is a Bangladeshi sports based romantic Movies. Its writer and director Khijir Hayat Khan. Adnan Karim sponsored youth football game built this film was released in 2010. The film is produced by Sharjeel Karim and Adnan Karim and directed by Khijir Hayat Khan who has written the story screenplay and dialogues of the film. The film features Ferdous Ahmed and Afsana Ara Bindu in lead roles and with supporting Arefin Shuvo Tariq Anam Ronok Hasaan and many more.","completion":"Film"}
|
||||
{"prompt":" John Edward Hatton AO (born 29 May 1933) is former Australian politician and an National Trust of Australia nominated Australian Living Treasure. He was the independent member of the Legislative Assembly of the New South Wales parliament for the seat of South Coast from 1973 to 1995. Notably the allegations about police corruption Hatton raised in Parliament resulted in the Wood Royal Commission. He is currently a social activist in his local community.","completion":"OfficeHolder"}
|
||||
{"prompt":" Trichoptilus subtilis is a moth of the Pterophoridae family that is known from South Africa.","completion":"Animal"}
|
||||
{"prompt":" Sin\u00e9ad Madden (born in Galway Ireland) is an Irish singer-songwriter and fiddle player best known as a member of the Moya Brennan band. She also teaches at Waltons New School of Music in Dublin.","completion":"Artist"}
|
||||
{"prompt":" Philip Sprint is a German footballer who currently plays for Hertha BSC.Sprint made his professional debut for Hertha BSC on 12 August 2012 in a 2. Bundesliga match against FSV Frankfurt coming on in the 50th minute for Marvin Knoll after starting goalkeeper Sascha Burchert had been sent off.","completion":"Athlete"}
|
||||
{"prompt":" River Roads Mall was an enclosed shopping mall located in the city of Jennings a suburb of St. Louis Missouri United States. Opened in 1962 as one of the nation's first shopping malls the mall declined in the 1990s becoming a dead mall and eventually being shuttered in 1995. Demolition of the long-vacant mall began in 2006.","completion":"Building"}
|
||||
{"prompt":" The Brown-patched Kangaroo lizard (Otocryptis wiegmanni) also called Wiegmann's Agama or Sri Lankan Kangaroo Lizard is a small ground dwelling agamid lizard endemic to the wet zone forests and lower mountain forests (up to 1300 metres) of Sri Lanka. It is commonly seen in the leaf litter of shady rain forests. When perceiving danger it spurts away quickly on its large hind legs and might eventually climb up a sapling or tree.","completion":"Animal"}
|
||||
{"prompt":" Shiho Kawaragi (\u6cb3\u539f\u6728 \u5fd7\u7a42 Kawaragi Shiho born April 29 1976 in Tokyo) is a Japanese voice actress who works for Kenyu-Office. When voicing adult games and hentai OVAs she is also known as Kaname Yuzuki (\u67da\u6728\u304b\u306a\u3081 Yuzuki Kaname) She is currently married since March 2012.","completion":"Artist"}
|
||||
{"prompt":" Down in the Shacks Where the Satellite Dishes Grow is the second album by the Judybats released in 1992.","completion":"Album"}
|
||||
{"prompt":" Turn of Faith is a 2001 film directed by Charles Jarrott. It stars Ray Mancini and Mia Sara.","completion":"Film"}
|
||||
{"prompt":" Frederick William Seward (July 8 1830 \u2013 April 25 1915) was the Assistant Secretary of State during the American Civil War serving in Abraham Lincoln's administration as well as under Andrew Johnson during Reconstruction and for over two years under Rutherford B. Hayes.","completion":"OfficeHolder"}
|
||||
{"prompt":" Ivoprop Corporation founded in 1984 by Ivo Zdarsky is an American manufacturer of composite propellers for homebuilt and ultralight aircraft as well as airboats. The company headquarters is located in Bellflower California.Zdarsky started the company after carving his own propeller for a homebuilt ultralight trike that he flew from Cold War Czechoslovakia over the Iron Curtain to Vienna in 1984.","completion":"Company"}
|
||||
{"prompt":" Wave Broadband is a provider of residential business and enterprise class cable TV broadband internet and telephone services on the West Coast currently serving about 400000 customers within communities in western Washington state Oregon Sacramento California and the San Francisco Bay Area. Wave Broadband provides services via their fiber-optic network and uses Northwest Open Access Network as the backbone for most of their service areas in Washington.","completion":"Company"}
|
||||
{"prompt":" Andie Tong is a comic book artist known for his work on books such as Spectacular Spider-Man UK The Batman Strikes! and Tangent: Superman's Reign.","completion":"Artist"}
|
||||
{"prompt":" Merdani is a village in the municipality of Busova\u010da Bosnia and Herzegovina.","completion":"Village"}
|
||||
{"prompt":" Kamam (Persian: \u0643\u0627\u0645\u0645\u200e also Romanized as K\u0101mam) is a village in Mangur-e Sharqi Rural District Khalifan District Mahabad County West Azerbaijan Province Iran. At the 2006 census its population was 98 in 16 families.","completion":"Village"}
|
||||
{"prompt":" Ficus greiffiana is a species of plant in the Moraceae family. It is found in Argentina Brazil Colombia and Guyana.","completion":"Plant"}
|
||||
{"prompt":" Toni Amboaje is a Spanish singer who currently works for metal band Sauze which formed on early 2008.","completion":"Artist"}
|
||||
{"prompt":" Mount Whittier is a mountain in Carroll County New Hampshire in the northern Ossipee Mountains. Named after John Greenleaf Whittier the peak is not to be confused with nearby Nickerson Mountain which was once known as Mount Whittier.There are no hiking trails on Mount Whittier. There was once a CCC alpine ski trail on the northern face.","completion":"NaturalPlace"}
|
||||
{"prompt":" El Rompe Discoteka: The Mix Album is a 2007 album by Hector El Father.","completion":"Album"}
|
||||
{"prompt":" e-Spirit is a commercial software company that develops and markets the FirstSpirit CMS Web content management system. The company was founded in 1999 in Dortmund Germany and established a US presence in 2011. The company's FirstSpirit CMS is a Java-based offering now in its fifth major release.[citation needed]","completion":"Company"}
|
||||
{"prompt":" The Valley is the first novel by Barry Pilton published in 2005 by Bloomsbury. It is a humorous account of the effect of outsiders on the rural status quo in a fictional mid-Wales valley during the 1980s and is being adapted for television.","completion":"WrittenWork"}
|
||||
{"prompt":" Sema Group plc was an Anglo-French IT services company. It was listed on the London Stock Exchange and was a constituent of the FTSE 100 Index but was acquired by Schlumberger in 2001.","completion":"Company"}
|
||||
{"prompt":" Bent Hansen (born 1954) is a retired Danish ice hockey forward. He played for 18 years in Denmark for the R\u00f8dovre SIK and KSF. He also competed for the Danish national team. His son Jannik Hansen also played for the R\u00f8dovre team and was drafted into the NHL by the Vancouver Canucks in 2004. During his hockey career Hansen also worked as a carpenter.","completion":"Athlete"}
|
||||
{"prompt":" Behind the Sun is a 2004 album by Dive.","completion":"Album"}
|
||||
{"prompt":" Mungaru Male (English: Pre Monsoon Rain) is a 2006 Kannada language movie directed by Yograj Bhat and produced by E Krishnappa. The film stars Ganesh Pooja Gandhi Anant Nag Padmaja Rao in lead roles.","completion":"Film"}
|
||||
{"prompt":" Megachile perihirta commonly known as the Western leafcutting bee is a bee in the genus Megachile. The bee is native to western North America ranging from Nebraska to Texas and Mexico west to California and north to British Columbia and Alberta and often inhabits meadows and orchards. The bee is black with long whitish-yellow hair more so below the thorax and abdomen. The abdomen however is mostly bare although each segment has scattered whitish hair.","completion":"Animal"}
|
||||
{"prompt":" The Maple School District is a public school district in Douglas County Wisconsin United States based in Maple Wisconsin.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Mount Waverley Secondary College is a public secondary school located in the Melbourne suburb of Mount Waverley. The school consists of roughly 1900 students and is one of the largest in the state.The school consists of two campuses (Junior & Senior) both situated on Stephensons Road in Mount Waverley. The Junior site holds years 7 and 8 with year levels 9 to 12 at the Senior Campus. The campuses are a short walking distance apart.","completion":"EducationalInstitution"}
|
||||
{"prompt":" Jon-Paul Roger JP Pietersen (born 12 July 1986 in Stellenbosch South Africa) is a South African rugby union footballer. He generally plays fullback or wing for the Sharks (in the Super Rugby competition) and the Natal Sharks in the Currie Cup. He played in more than 50 tests for the Springboks.","completion":"Athlete"}
|
||||
{"prompt":" Deltocolpodes is a genus of beetles in the family Carabidae containing the following species: Deltocolpodes brendelli Morvan 1992 Deltocolpodes championi Morvan 1992 Deltocolpodes duluchus Morvan 1992 Deltocolpodes heinigeri Morvan 1992 Deltocolpodes jalepensis Morvan 1992 Deltocolpodes kirschenhoferi Morvan 1992 Deltocolpodes nepalensis Morvan 1992 Deltocolpodes perreaui Deuve 1985 Deltocolpodes rectangulus Morvan 1992 Deltocolpodes rolex Morvan 1992 Deltocolpodes salpensis Deuve 1985 Deltocolpodes sikkimensis Morvan 1992\u2191","completion":"Animal"}
|
||||
{"prompt":" Stanhopea martiana is a species of orchid endemic to southwestern Mexico.","completion":"Plant"}
|
||||
{"prompt":" Yawarmayu (Quechua yawar blood mayu river blood river hispanicized spelling Yahuarmayo) is a river in Peru located in the Puno Region Carabaya Province Ayapata District. It originates near the border of the districts Ayapata and Coasa. Its direction is mainly to the northwest where it meets Inambari River as a right affluent. The confluence is north of the village Yawarmayu (Yahuarmayo).","completion":"NaturalPlace"}
|
||||
{"prompt":" The Charles Granke House at 406 S. Seventh St. in Hamilton Montana is a historic house that was built in 1906. It includes Colonial Revival and Queen Anne architecture. It was listed on the National Register of Historic Places in 1988. The listing included two contributing buildings.It was built in approximately 1906 by the Anaconda Copper Mining Company as a worker cottage for workers at the sawmill that operated in Hamilton until 1915. Charles W.","completion":"Building"}
|
||||
{"prompt":" Passiflora monadelpha is a species of plant in the Passifloraceae family. It is endemic to Ecuador.","completion":"Plant"}
|
||||
{"prompt":" Mangifera persiciformis or Peach Mango is a species of plant in the Anacardiaceae family. It is endemic to China.","completion":"Plant"}
|
||||
{"prompt":"The CN Tower (French: Tour CN) is a 553.3 m-high (1,815.3 ft) concrete communications and observation tower in downtown Toronto, Ontario, Canada.","completion":"Building"}
|
||||
|
||||
@ -1,102 +0,0 @@
|
||||
Date,Supplier,Description,Transaction value (£),Classification
|
||||
15/08/2016,Creative Video Productions Ltd,Kelvin Hall,26866,Other
|
||||
29/05/2017,John Graham Construction Ltd,Causewayside Refurbishment,74806,Building Improvement
|
||||
29/05/2017,Morris & Spottiswood Ltd,George IV Bridge Work,56448,Building Improvement
|
||||
31/05/2017,John Graham Construction Ltd,Causewayside Refurbishment,164691,Building Improvement
|
||||
24/07/2017,John Graham Construction Ltd,Causewayside Refurbishment,27926,Building Improvement
|
||||
24/07/2017,John Graham Construction Ltd,Causewayside Refurbishment,212690,Building Improvement
|
||||
16/08/2017,John Graham Construction Ltd,Causewayside Refurbishment,59021,Building Improvement
|
||||
16/08/2017,John Graham Construction Ltd,Causewayside Refurbishment,136379,Building Improvement
|
||||
23/08/2017,Culture And Sport Glasgow,Kelvin Hall,60503,Building Improvement
|
||||
23/08/2017,XMA Scotland Ltd,Kelvin Hall,31830,Building Improvement
|
||||
31/08/2017,John Graham Construction Ltd,Causewayside Refurbishment,36313,Building Improvement
|
||||
31/08/2017,Insight Direct (UK) Ltd,Causewayside Refurbishment,68222,Building Improvement
|
||||
31/08/2017,Mark Finn Laboratory,George IV Bridge Work,53884,Building Improvement
|
||||
11/09/2017,John Graham Construction Ltd,Causewayside Refurbishment,189483,Building Improvement
|
||||
23/10/2017,John Graham Construction Ltd,Causewayside Refurbishment,151659,Building Improvement
|
||||
23/10/2017,City Building LLP,Causewayside Refurbishment,53147,Building Improvement
|
||||
07/02/2017,John Graham Construction Ltd,Causewayside Refurbishment,52404,Building Improvement
|
||||
13/02/2017,John Graham Construction Ltd,Causewayside Refurbishment,272390,Building Improvement
|
||||
06/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,31781,Building Improvement
|
||||
06/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,198048,Building Improvement
|
||||
31/03/2017,Nicholson Bros(Electrical Contractors) Ltd,Causewayside Refurbishment,33666,Building Improvement
|
||||
31/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,222090,Building Improvement
|
||||
31/03/2017,John Graham Construction Ltd,Causewayside Refurbishment,63971,Building Improvement
|
||||
24/04/2017,Scottish Historic Buildings Trust,Lawnmarket Work,50057,Building Improvement
|
||||
30/04/2017,Morris & Spottiswood Ltd,George IV Bridge Work,63716,Building Improvement
|
||||
15/05/2017,John Graham Construction Ltd,Causewayside Refurbishment,245381,Building Improvement
|
||||
12/09/2016,Flexiform,Kelvin Hall,42623,Building Improvement
|
||||
12/09/2016,John Graham Construction Ltd,Causewayside Refurbishment,228689,Building Improvement
|
||||
26/09/2016,Senator International,Kelvin Hall,35706,Building Improvement
|
||||
26/09/2016,John Graham Construction Ltd,Causewayside Refurbishment,28378,Building Improvement
|
||||
30/09/2016,A McGillivray,Causewayside Refurbishment,44392,Building Improvement
|
||||
10/10/2016,John Graham Construction Ltd,Causewayside Refurbishment,303999,Building Improvement
|
||||
31/10/2016,John Graham Construction Ltd,Causewayside Refurbishment,74245,Building Improvement
|
||||
07/11/2016,CBRE,Kelvin Hall,83736,Building Improvement
|
||||
14/11/2016,University Of Glasgow,Kelvin Hall,188682,Building Improvement
|
||||
14/11/2016,John Graham Construction Ltd,Causewayside Refurbishment,362326,Building Improvement
|
||||
12/12/2016,John Graham Construction Ltd,Causewayside Refurbishment,385310,Building Improvement
|
||||
30/12/2016,John Graham Construction Ltd,Causewayside Refurbishment,253618,Building Improvement
|
||||
30/12/2016,John Graham Construction Ltd,Causewayside Refurbishment,45127,Building Improvement
|
||||
21/04/2016,M & J Ballantyne Ltd,George IV Bridge Work,35098,Building Improvement
|
||||
09/05/2016,John Graham Construction Ltd,Causewayside Refurbishment,64361,Building Improvement
|
||||
09/05/2016,A McGillivray,Causewayside Refurbishment,53690,Building Improvement
|
||||
16/05/2016,John Graham Construction Ltd,Causewayside Refurbishment,365344,Building Improvement
|
||||
10/06/2016,Wavetek Ltd,Kelvin Hall,87589,Building Improvement
|
||||
10/06/2016,John Graham Construction Ltd,Causewayside Refurbishment,381803,Building Improvement
|
||||
30/06/2016,Glasgow City Council,Kelvin Hall,1700000,Building Improvement
|
||||
11/07/2016,Wavetek Ltd,Kelvin Hall,65692,Building Improvement
|
||||
11/07/2016,John Graham Construction Ltd,Causewayside Refurbishment,139845,Building Improvement
|
||||
25/07/2016,A McGillivray,Causewayside Refurbishment,30113,Building Improvement
|
||||
15/08/2016,John Graham Construction Ltd,Causewayside Refurbishment,196807,Building Improvement
|
||||
06/11/2017,John Graham Construction Ltd,Causewayside Refurbishment,134208,Building Improvement
|
||||
31/03/2017,NLS Foundation,Grant Payment,177500,Other
|
||||
09/10/2017,Frost And Sullivan Ltd,Literary & Archival Items,28125,Literature & Archive
|
||||
09/10/2017,JISC Services Ltd ,Literary & Archival Items,43481,Literature & Archive
|
||||
27/02/2017,Cengage Learning (Emea )Ltd,Literary & Archival Items,43302,Literature & Archive
|
||||
06/03/2017,Private Sale,Literary & Archival Items,72500,Literature & Archive
|
||||
31/03/2017,Private Sale,Literary & Archival Items,3422500,Literature & Archive
|
||||
24/04/2017,Cengage Learning (Emea )Ltd,Literary & Archival Items,43302,Literature & Archive
|
||||
22/05/2017,ALDL,Legal Deposit Services,27067,Literature & Archive
|
||||
19/09/2016,Jisc Services Ltd Subscription Account,Literary & Archival Items,42629,Literature & Archive
|
||||
10/10/2016,Cengage Learning (Emea )Ltd,Literary & Archival Items,86604,Literature & Archive
|
||||
24/10/2016,ALDL,ALDL Charges,32317,Literature & Archive
|
||||
26/04/2016,Private Sale,Literary & Archival Items,30000,Literature & Archive
|
||||
30/05/2016,ALDL,ALDL Charges,32317,Literature & Archive
|
||||
15/07/2016,Sotheby'S,Literary & Archival Items,28500,Literature & Archive
|
||||
18/07/2016,Christies,Literary & Archival Items,33800,Literature & Archive
|
||||
31/07/2016,ALDL,ALDL Charges,32317,Literature & Archive
|
||||
08/12/2016,Sothebys,Literary & Archival Items,166000,Literature & Archive
|
||||
08/12/2016,Private Sale,Literary & Archival Items,87500,Literature & Archive
|
||||
26/06/2017,ECG Facilities Service,Facilities Management Charge,33386,Utility Bills
|
||||
26/06/2017,British Library,Legal Deposit Services,50056,Other
|
||||
24/07/2017,ALDL,Legal Deposit Services,27067,Other
|
||||
16/08/2017,ECG Facilities Service,Facilities Management Charge,33386,Utility Bills
|
||||
23/08/2017,ECG Facilities Service,Facilities Management Charge,33386,Utility Bills
|
||||
07/02/2017,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
27/02/2017,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
27/03/2017,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
22/05/2017,ECG Facilities Service,Facilities Management Charge,33386,Utility Bills
|
||||
26/09/2016,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
24/10/2016,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
08/12/2016,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
30/12/2016,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
23/05/2016,ECG Facilities Service,Facilities Management Charge,32777,Utility Bills
|
||||
23/05/2016,ECG Facilities Service,Facilities Management Charge,32777,Utility Bills
|
||||
28/06/2016,ECG Facilities Service,Facilities Management Charge,32832,Utility Bills
|
||||
08/08/2016,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
24/08/2016,ECG Facilities Service,Facilities Management Charge,32795,Utility Bills
|
||||
30/10/2017,ECG Facilities Service,Facilities Management Charge,35758,Utility Bills
|
||||
16/08/2017,Ex Libris,IT equipment,76610,Software/IT
|
||||
31/03/2017,XMA Scotland Ltd,IT equipment,33450,Software/IT
|
||||
31/03/2017,XMA Scotland Ltd,IT equipment,84524,Software/IT
|
||||
24/04/2017,Insight Direct (UK) Ltd,IT equipment,56768,Software/IT
|
||||
09/05/2016,Computacenter Uk,Kelvin Hall,72835,Software/IT
|
||||
23/05/2016,Computacenter Uk,Kelvin Hall,26506,Software/IT
|
||||
15/09/2017,City Of Edinburgh Council,Non Domestic Rates ,57662,Utility Bills
|
||||
15/09/2017,City Of Edinburgh Council,Non Domestic Rates ,142680,Utility Bills
|
||||
08/05/2017,Anglian Water Business,Water,26832,Utility Bills
|
||||
30/04/2016,City Of Edinburgh Council,Non Domestic Rates ,40800,Utility Bills
|
||||
12/09/2016,City Of Edinburgh Council,Non Domestic Rates ,144330,Utility Bills
|
||||
12/09/2016,City Of Edinburgh Council,Non Domestic Rates ,49827,Utility Bills
|
||||
24/07/2017,AM Phillip,Vehicle Purchase,26604,Other
|
||||
|
@ -1,9 +1,3 @@
|
||||
"""
|
||||
Note: To answer questions based on text documents, we recommend the procedure in
|
||||
[Question Answering using Embeddings](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb).
|
||||
Some of the code below may rely on [deprecated API endpoints](https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
||||
import openai
|
||||
|
||||
@ -1,12 +1,5 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<span style=\"color:orange; font-weight:bold\">Note: To answer questions based on text documents, we recommend the procedure in <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering using Embeddings</a>. Some of the code below may rely on <a href=\"https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints\">deprecated API endpoints</a>.</span>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
@ -171,7 +164,7 @@
|
||||
" discard_categories: Set[str] = discard_categories,\n",
|
||||
") -> str:\n",
|
||||
" \"\"\"\n",
|
||||
" Extract the sections of a Wikipedia page, discarding the references and other low information sections\n",
|
||||
" Extract the sections of a Wikipedia page, discarding the the references and other low information sections\n",
|
||||
" \"\"\"\n",
|
||||
" if len(wiki_text) == 0:\n",
|
||||
" return []\n",
|
||||
@ -494,9 +487,11 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "be4b5d5b73a21c599de40d6deb1129796d12dc1cc33a738f7bac13269cfcafe8"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.9 64-bit ('3.9.9')",
|
||||
"language": "python",
|
||||
"display_name": "Python 3.7.3 64-bit ('base': conda)",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
@ -509,14 +504,9 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.9"
|
||||
"version": "3.7.3"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "cb9817b186a29e4e9713184d901f26c1ee05ad25243d878baff7f31bb1fef480"
|
||||
}
|
||||
}
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
|
||||
@ -1,12 +1,5 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<span style=\"color:orange; font-weight:bold\">Note: To answer questions based on text documents, we recommend the procedure in <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering using Embeddings</a>. Some of the code below may rely on <a href=\"https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints\">deprecated API endpoints</a>.</span>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
@ -149,7 +142,7 @@
|
||||
"\n",
|
||||
"Note: We have used temperature=0, but it may be beneficial to experiment with a higher temperature to get a higher diversity of questions.\n",
|
||||
"\n",
|
||||
"<span style=\"color:orange; font-weight:bold\">WARNING: This step will last a long time, and consume a lot of tokens, as it calls davinci-instruct for every section to generate a number of questions.</span>"
|
||||
"<span style=\"color:orange\">**WARNING: This step will last a long time, and consume a lot of tokens, as it calls davinci-instruct for every section to generate a number of questions.**</span>"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -308,7 +301,7 @@
|
||||
"## 2.5 Search file (DEPRECATED)\n",
|
||||
"We create a search file ([API reference](https://beta.openai.com/docs/api-reference/files/list)), which can be used to retrieve the relevant context when a question is asked.\n",
|
||||
"\n",
|
||||
"<span style=\"color:orange; font-weight:bold\">DEPRECATED: The /search endpoint is deprecated in favour of using embeddings. Embeddings are cheaper, faster and can support a better search experience. See <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering Guide</a> for a search implementation using the embeddings</span>\n"
|
||||
"<span style=\"color:orange\">**DEPRECATED: The /search endpoint is deprecated in favour of using embeddings. Embeddings are cheaper, faster and can support a better search experience. See [Question Answering Guide](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb) for a search implementation using the embeddings**</span>\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@ -1,12 +1,5 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<span style=\"color:orange; font-weight:bold\">Note: To answer questions based on text documents, we recommend the procedure in <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering using Embeddings</a>. Some of the code below may rely on <a href=\"https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints\">deprecated API endpoints</a>.</span>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
@ -618,9 +611,11 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "be4b5d5b73a21c599de40d6deb1129796d12dc1cc33a738f7bac13269cfcafe8"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.9 64-bit ('3.9.9')",
|
||||
"language": "python",
|
||||
"display_name": "Python 3.7.3 64-bit ('base': conda)",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
@ -633,14 +628,9 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.9"
|
||||
"version": "3.7.3"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "cb9817b186a29e4e9713184d901f26c1ee05ad25243d878baff7f31bb1fef480"
|
||||
}
|
||||
}
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 799 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 301 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 355 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 192 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 332 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 160 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 101 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 298 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 133 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 108 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 92 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 81 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 303 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 64 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 56 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 57 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 126 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 198 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 175 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 855 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 487 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 440 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 692 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 340 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 273 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 216 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 281 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.8 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 362 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 323 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 305 KiB |
@ -1,574 +0,0 @@
|
||||
# Techniques to improve reliability
|
||||
|
||||
When GPT-3 fails on a task, what should you do?
|
||||
|
||||
- Search for a better prompt that elicits more reliable answers?
|
||||
- Invest in thousands of examples to fine-tune a custom model?
|
||||
- Assume the model is incapable of the task, and move on?
|
||||
|
||||
There is no simple answer - it depends. However, if your task involves logical reasoning or complexity, consider trying the techniques in this article to build more reliable, high-performing prompts.
|
||||
|
||||
## Why GPT-3 fails on complex tasks
|
||||
|
||||
If you were asked to multiply 13 by 17, would the answer pop immediately into your mind? For most of us, probably not. Yet, that doesn't mean humans are incapable of two-digit multiplication. With a few seconds, and some pen and paper, it's not too taxing to work out that 13 x 17 = 130 + 70 + 21 = 221.
|
||||
|
||||
Similarly, if you give GPT-3 a task that's too complex to do in the time it takes to calculate its next token, it may confabulate an incorrect guess. Yet, akin to humans, that doesn't necessarily mean the model is incapable of the task. With some time and space to reason things out, the model still may be able to answer reliably.
|
||||
|
||||
As an example, if you ask `text-davinci-002` the following math problem about juggling balls, it answers incorrectly:
|
||||
|
||||
```text-davinci-002
|
||||
Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there?
|
||||
A:
|
||||
```
|
||||
|
||||
```text-davinci-002
|
||||
There are 8 blue golf balls.
|
||||
```
|
||||
|
||||
Does this mean that GPT-3 cannot do simple math problems? No; in fact, it turns out that by prompting the model with `Let's think step by step`, the model solves the problem reliably:
|
||||
|
||||
```text-davinci-002
|
||||
Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there?
|
||||
A: Let's think step by step.
|
||||
```
|
||||
|
||||
```text-davinci-002
|
||||
There are 16 balls in total.
|
||||
Half of the balls are golf balls.
|
||||
That means that there are 8 golf balls.
|
||||
Half of the golf balls are blue.
|
||||
That means that there are 4 blue golf balls.
|
||||
```
|
||||
|
||||
Of course, it's hard to tell from only a single example whether this `Let's think step by step` trick actually works in general or just got lucky on this particular problem. But it really does work. On a benchmark of word math problems, the `Let's think step by step` trick raised GPT-3's solve rate massively, from a worthless 18% to a decent 79%!
|
||||
|
||||
## Model capabilities depend on context
|
||||
|
||||
When learning to work with GPT-3, one common conceptual mistake is to believe that its capabilities are fixed across all contexts. E.g., if GPT-3 gets a simple logic question wrong, then it must be incapable of simple logic.
|
||||
|
||||
But as the `Let's think step by step` example illustrates, apparent failures of GPT-3 can sometimes be remedied with a better prompt that helps the model steer itself toward the correct output.
|
||||
|
||||
## How to improve reliability on complex tasks
|
||||
|
||||
The rest of this article shares techniques for improving reliability of large language models on complex tasks. Although some of the techniques are specific to certain types of problems, many of them are built upon general principles that can be applied to a wide range of tasks, e.g.:
|
||||
|
||||
- Give clearer instructions
|
||||
- Split complex tasks into simpler subtasks
|
||||
- Structure the instruction to keep the model on task
|
||||
- Prompt the model to explain before answering
|
||||
- Ask for justifications of many possible answers, and then synthesize
|
||||
- Generate many outputs, and then use the model to pick the best one
|
||||
- Fine-tune custom models to maximize performance
|
||||
|
||||
## Split complex tasks into simpler tasks
|
||||
|
||||
One way to give a model more time and space to think is to break tasks into simpler pieces.
|
||||
|
||||
As an example, consider a task where we ask the model a multiple-choice question about some text - in this case, a game of Clue. When asked directly, `text-davinci-002` isn't able to put clues 3 & 5 together, and answers incorrectly:
|
||||
|
||||
```text-davinci-002
|
||||
Use the following clues to answer the following multiple-choice question.
|
||||
|
||||
Clues:
|
||||
1. Miss Scarlett was the only person in the lounge.
|
||||
2. The person with the pipe was in the kitchen.
|
||||
3. Colonel Mustard was the only person in the observatory.
|
||||
4. Professor Plum was not in the library nor the billiard room.
|
||||
5. The person with the candlestick was in the observatory.
|
||||
|
||||
Question: Was Colonel Mustard in the observatory with the candlestick?
|
||||
(a) Yes; Colonel Mustard was in the observatory with the candlestick
|
||||
(b) No; Colonel Mustard was not in the observatory with the candlestick
|
||||
(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick
|
||||
|
||||
Solution:
|
||||
```
|
||||
|
||||
```text-davinci-002
|
||||
(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick
|
||||
```
|
||||
|
||||
Although clues 3 and 5 establish that Colonel Mustard was the only person in the observatory and that the person in the observatory had the candlestick, the models fails to combine them into a correct answer of (a) Yes.
|
||||
|
||||
However, instead of asking for the answer directly, we can split the task into three pieces:
|
||||
|
||||
- First, go through the clues one by one and and consider whether the clue is potentially relevant
|
||||
- Second, combine the relevant clues to reason out the answer to the question
|
||||
- Third, write the final answer: either (a), (b), or (c)
|
||||
|
||||
```text-davinci-002
|
||||
Use the following clues to answer the following multiple-choice question, using the following procedure:
|
||||
(1) First, go through the clues one by one and and consider whether the clue is potentially relevant
|
||||
(2) Second, combine the relevant clues to reason out the answer to the question
|
||||
(3) Third, map the answer to one of the multiple choice answers: either (a), (b), or (c)
|
||||
|
||||
Clues:
|
||||
1. Miss Scarlett was the only person in the lounge.
|
||||
2. The person with the pipe was in the kitchen.
|
||||
3. Colonel Mustard was the only person in the observatory.
|
||||
4. Professor Plum was not in the library nor the billiard room.
|
||||
5. The person with the candlestick was in the observatory.
|
||||
|
||||
Question: Was Colonel Mustard in the observatory with the candlestick?
|
||||
(a) Yes; Colonel Mustard was in the observatory with the candlestick
|
||||
(b) No; Colonel Mustard was not in the observatory with the candlestick
|
||||
(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick
|
||||
|
||||
Solution:
|
||||
(1) First, go through the clues one by one and and consider whether the clue is potentially relevant:
|
||||
```
|
||||
|
||||
```text-davinci-002
|
||||
Clue 1: Miss Scarlett was the only person in the lounge. This clue is not relevant.
|
||||
Clue 2: The person with the pipe was in the kitchen. This clue is not relevant.
|
||||
Clue 3: Colonel Mustard was the only person in the observatory. This clue is relevant.
|
||||
Clue 4: Professor Plum was not in the library nor the billiard room. This clue is not relevant.
|
||||
Clue 5: The person with the candlestick was in the observatory. This clue is relevant.
|
||||
|
||||
(2) Second, combine the relevant clues to reason out the answer to the question:
|
||||
Clue 3 and Clue 5 both say that the person with the candlestick was in the observatory. Colonel Mustard was the only person in the observatory, so Colonel Mustard was the person with the candlestick.
|
||||
|
||||
(3) Third, write the final answer:
|
||||
(a) Yes; Colonel Mustard was in the observatory with the candlestick
|
||||
```
|
||||
|
||||
By giving the model more time and space to think, and guiding it along a reasoning plan, it's able to figure out the correct answer of (a) Yes.
|
||||
|
||||
Another benefit of splitting complex instructions into smaller subtasks is that it can help keep the model focused on each subtask.
|
||||
|
||||
For example, if we ask `text-davinci-002` to summarize a text in its original language, the model can lapse back into English:
|
||||
|
||||
```text-davinci-002
|
||||
Summarize the text using the original language of the text. The summary should be one sentence long.
|
||||
|
||||
Text:
|
||||
"""
|
||||
La estadística (la forma femenina del término alemán Statistik, derivado a su vez del italiano statista, "hombre de Estado") es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad. La estadística es una ciencia formal deductiva, con un conocimiento propio, dinámico y en continuo desarrollo obtenido a través del método científico formal. En ocasiones, las ciencias fácticas necesitan utilizar técnicas estadísticas durante su proceso de investigación factual, con el fin de obtener nuevos conocimientos basados en la experimentación y en la observación. En estos casos, la aplicación de la estadística permite el análisis de datos provenientes de una muestra representativa, que busca explicar las correlaciones y dependencias de un fenómeno físico o natural, de ocurrencia en forma aleatoria o condicional.
|
||||
"""
|
||||
|
||||
Summary:
|
||||
```
|
||||
|
||||
```text-davinci-002
|
||||
The text explains that statistics is a science that studies the variability, collection, organization, analysis, interpretation, and presentation of data, as well as the random process that generates them following the laws of probability.
|
||||
```
|
||||
|
||||
However, if we first ask the model to identify the language of the text, and then summarize the text, it becomes more reliable:
|
||||
|
||||
```text-davinci-002
|
||||
First, identify the language of the text. Second, summarize the text using the original language of the text. The summary should be one sentence long.
|
||||
|
||||
Text:
|
||||
"""
|
||||
La estadística (la forma femenina del término alemán Statistik, derivado a su vez del italiano statista, "hombre de Estado") es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad. La estadística es una ciencia formal deductiva, con un conocimiento propio, dinámico y en continuo desarrollo obtenido a través del método científico formal. En ocasiones, las ciencias fácticas necesitan utilizar técnicas estadísticas durante su proceso de investigación factual, con el fin de obtener nuevos conocimientos basados en la experimentación y en la observación. En estos casos, la aplicación de la estadística permite el análisis de datos provenientes de una muestra representativa, que busca explicar las correlaciones y dependencias de un fenómeno físico o natural, de ocurrencia en forma aleatoria o condicional.
|
||||
"""
|
||||
|
||||
Language:
|
||||
```
|
||||
|
||||
```text-davinci-002
|
||||
Spanish
|
||||
|
||||
La estadística es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad.
|
||||
```
|
||||
|
||||
## Prompt the model to explain before answering
|
||||
|
||||
Another powerful technique for improving the reliability of answers is to prompt the model to gradually reason out the answer rather than jumping immediately to the final answer. By 'thinking aloud' the model can be far more likely to arrive at the correct answer.
|
||||
|
||||
### Zero-shot
|
||||
|
||||
#### Method
|
||||
|
||||
Published by [Takeshi Kojima et al. in 2022](https://arxiv.org/abs/2205.11916), the easiest way to prompt a model to reason out the answer is to simply prepend answers with `Let's think step by step.` Figure 2 illustrates an example:
|
||||
|
||||
[
|
||||
<br>Source: *Large Language Models are Zero-Shot Reasoners* by Takeshi Kojima et al. (2022).](https://arxiv.org/abs/2205.11916)
|
||||
|
||||
#### Results
|
||||
|
||||
Applying this simple trick to the MultiArith math dataset, the authors found `Let's think step by step` quadrupled the accuracy, from 18% to 79%!
|
||||
|
||||
[
|
||||
<br>Source: *Large Language Models are Zero-Shot Reasoners* by Takeshi Kojima et al. (2022).](https://arxiv.org/abs/2205.11916)
|
||||
|
||||
#### Implications
|
||||
|
||||
Although the `Let's think step by step` trick works well on math problems, it's not effective on all tasks. The authors found that it was most helpful for multi-step arithmetic problems, symbolic reasoning problems, strategy problems, and other reasoning problems. It didn't help with simple math problems or common sense questions, and presumably wouldn't help with many other non-reasoning tasks either.
|
||||
|
||||
[
|
||||
<br>Source: *Large Language Models are Zero-Shot Reasoners* by Takeshi Kojima et al. (2022).](https://arxiv.org/abs/2205.11916)
|
||||
|
||||
To learn more, read the [full paper](https://arxiv.org/abs/2205.11916).
|
||||
|
||||
If you apply this technique to your own tasks, don't be afraid to experiment with customizing the instruction. `Let's think step by step` is rather generic, so you may find better performance with instructions that hew to a stricter format customized to your use case. For example, if you were you can try more structured variants like `First, think step by step about why X might be true. Second, think step by step about why Y might be true. Third, think step by step about whether X or Y makes more sense.`. And you can even give the model an example format to help keep it on track, e.g.:
|
||||
|
||||
```text-davinci-002
|
||||
Using the IRS guidance below, answer the following questions using this format:
|
||||
(1) For each criterion, determine whether it is met by the vehicle purchase
|
||||
- {Criterion} Let's think step by step. {explanation} {yes or no, or if the question does not apply then N/A}.
|
||||
(2) After considering each criterion in turn, phrase the final answer as "Because of {reasons}, the answer is likely {yes or no}."
|
||||
|
||||
IRS guidance:
|
||||
"""
|
||||
You may be eligible for a federal tax credit under Section 30D if you purchased a car or truck that meets the following criteria:
|
||||
- Does the vehicle have at least four wheels?
|
||||
- Does the vehicle weigh less than 14,000 pounds?
|
||||
- Does the vehicle draw energy from a battery with at least 4 kilowatt hours that may be recharged from an external source?
|
||||
- Was the vehicle purchased in a year before 2022?
|
||||
- If so, has the manufacturer sold less than 200,000 qualifying vehicles? (Tesla and GM have sold more than 200,000 qualifying vehicles.)
|
||||
- Was the vehicle purchased in a year after 2022?
|
||||
- If so, is the vehicle present in the following list of North American-assembled vehicles? (The only electric vehicles assembled in North America are the Audi Q5, BMW 330e, BMW X5, Chevrolet Bolt EUV, Chevrolet Bolt EV, Chrysler Pacifica PHEV, Ford Escape PHEV, Ford F Series, Ford Mustang MACH E, Ford Transit Van, GMC Hummer Pickup, GMC Hummer SUV, Jeep Grand Cherokee PHEV, Jeep Wrangler PHEV, Lincoln Aviator PHEV, Lincoln Corsair Plug-in, Lucid Air, Nissan Leaf, Rivian EDV, Rivian R1S, Rivian R1T, Tesla Model 3, Tesla Model S, Tesla Model X, Tesla Model Y, Volvo S60, BMW 330e, Bolt EV, Cadillac Lyriq, Mercedes EQS SUV, and Nissan Leaf.)
|
||||
"""
|
||||
|
||||
Question: Can I claim a federal tax credit for my Toyota Prius Prime bought in 2021?
|
||||
|
||||
Solution:
|
||||
|
||||
(1) For each criterion, determine whether it is met by the vehicle purchase
|
||||
- Does the vehicle have at least four wheels? Let's think step by step.
|
||||
```
|
||||
|
||||
```text-davinci-002
|
||||
The Toyota Prius Prime has four wheels, so the answer is yes.
|
||||
- Does the vehicle weigh less than 14,000 pounds? Let's think step by step. The Toyota Prius Prime weighs less than 14,000 pounds, so the answer is yes.
|
||||
- Does the vehicle draw energy from a battery with at least 4 kilowatt hours that may be recharged from an external source? Let's think step by step. The Toyota Prius Prime has a battery with at least 4 kilowatt hours that may be recharged from an external source, so the answer is yes.
|
||||
- Was the vehicle purchased in a year before 2022? Let's think step by step. The Toyota Prius Prime was purchased in 2021, which is before 2022, so the answer is yes.
|
||||
- Was the vehicle purchased in a year after 2022? N/A
|
||||
- If so, is the vehicle present in the following list of North American-assembled vehicles? N/A
|
||||
|
||||
(2) After considering each criterion in turn, phrase the final answer as "Because of {reasons}, the answer is likely {yes or no}."
|
||||
|
||||
Because the Toyota Prius Prime meets all of the criteria for a federal tax credit, the answer is likely yes.
|
||||
```
|
||||
|
||||
### Few-shot examples
|
||||
|
||||
#### Method
|
||||
|
||||
Prompting the model to reason out its answers can be done in many ways. One way is to demonstrate with a few examples ('few-shot'), as studied by [Jason Wei and Denny Zhou et al. from Google](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html). Here's an example few-shot chain-of-thought prompt:
|
||||
|
||||
[
|
||||
<br>Source: *Chain of Thought Prompting Elicits Reasoning in Large Language Models* Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
|
||||
|
||||
More demonstrations of reasoning chains written by human labelers:
|
||||
|
||||
[
|
||||
<br>Source: *Chain of Thought Prompting Elicits Reasoning in Large Language Models* Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
|
||||
|
||||
[(Note that it has been called into question whether pears actually float)](https://twitter.com/Meaningness/status/1561062170074370048?s=20&t=mpHt8f3RRboztXxdhLFnWQ)
|
||||
|
||||
#### Results
|
||||
|
||||
Testing on grade school math problems, the authors found that chain of thought prompting tripled the solve rate, from 18% to 57%.
|
||||
|
||||
[
|
||||
<br>Source: *Chain of Thought Prompting Elicits Reasoning in Large Language Models* Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
|
||||
|
||||
In addition to math problems, chain of thought prompting also lifted performance on questions related to sports understanding, coin flip tracking, and last letter concatenation. In most cases, not many examples were need to saturate the performance gains (less than 8 or so).
|
||||
|
||||
[
|
||||
<br>Source: *Chain of Thought Prompting Elicits Reasoning in Large Language Models* Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
|
||||
|
||||
To learn more, read the [full paper](https://arxiv.org/abs/2201.11903).
|
||||
|
||||
#### Implications
|
||||
|
||||
One advantage of the few-shot example-based approach relative to the `Let's think step by step` technique is that you can more easily specify the format, length, and style of reasoning that you want the model to perform before landing on its final answer. This can be be particularly helpful in cases where the model isn't initially reasoning in the right way or depth.
|
||||
|
||||
### Fine-tuned
|
||||
|
||||
#### Method
|
||||
|
||||
In general, to eke out maximum performance on a task, you'll need to fine-tune a custom model. However, fine-tuning a model using explanations may take thousands of example explanations, which are costly to write.
|
||||
|
||||
In 2022, Eric Zelikman and Yuhuai Wu et al. published a clever procedure for using a few-shot prompt to generate a dataset of explanations that could be used to fine-tune a model. The idea is to use a few-shot prompt to generate candidate explanations, and only keep the explanations that produce the correct answer. Then, to get additional explanations for some of the incorrect answers, retry the the few-shot prompt but with correct answers given as part of the question. The authors called their procedure STaR (Self-taught Reasoner):
|
||||
|
||||
[
|
||||
<br>Source: *STaR: Bootstrapping Reasoning With Reasoning* by Eric Zelikman and Yujuai Wu et al. (2022)](https://arxiv.org/abs/2203.14465)
|
||||
|
||||
With this technique, you can combine the benefits of fine-tuning with the benefits of chain-of-thought prompting without needing to write thousands of example explanations.
|
||||
|
||||
#### Results
|
||||
|
||||
When the authors applied this technique to a Common Sense Q&A dataset, they found that STaR outperformed both chain-of-thought prompting alone (73% > 37%) and fine-tuning alone (73% > 60%):
|
||||
|
||||
[
|
||||
<br>Source: *STaR: Bootstrapping Reasoning With Reasoning* by Eric Zelikman and Yujuai Wu et al. (2022)](https://arxiv.org/abs/2203.14465)
|
||||
|
||||
To learn more, read the [full paper](https://arxiv.org/abs/2203.14465).
|
||||
|
||||
#### Implications
|
||||
|
||||
Using a few-shot prompt to extend or modify a fine-tuning dataset is an idea that can be generalized beyond explanation writing. For example, if you have large quantities of unstructured text that you want to train on, you may find opportunities to use a prompt to extract a structured dataset from your unstructured text, and then fine-tune a custom model on that structured dataset.
|
||||
|
||||
## Extensions to chain-of-thought prompting
|
||||
|
||||
A number of extensions of chain-of-thought prompting have been published as well.
|
||||
|
||||
### Selection-inference prompting
|
||||
|
||||
#### Method
|
||||
|
||||
Published by Antonia Creswell et al., one extension of the chain-of-thought technique is to split the single prompt for generating explanations and answers into smaller parts. First, a prompt selects a relevant subset of facts from the text ('selection prompt'). Then, a second prompt infers a conclusion from the selected facts ('inference prompt'). These prompts are then alternated in a loop to generate multiple steps of reasoning and eventually land on a final answer. The authors illustrate the idea in the following figure:
|
||||
|
||||
[
|
||||
<br>Source: *Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2205.09712)
|
||||
|
||||
#### Results
|
||||
|
||||
When applied to a 7B-parameter model, the authors found that selection-inference prompting substantially improved performance relative to chain-of-thought prompting on the bAbi and Proof Writer benchmark tasks (both of which require longer sequences of reasoning steps). The best performance they achieved combined both selection-inference prompting with fine-tuning.
|
||||
|
||||
[
|
||||
<br>Source: *Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2205.09712)
|
||||
|
||||
#### Implications
|
||||
|
||||
Although the gains on these benchmarks were large, these benchmarks were specifically chosen because they required longer sequences of reasoning. On problems that don't require reasoning with many steps, the gains are likely smaller.
|
||||
|
||||
The results highlight a couple of general lessons for working with large language models. One, splitting up complex tasks into smaller tasks is a great way to improve reliability and performance; the more atomic the task, the less room there is for the model to err. Two, getting maximum performance often means combining fine-tuning with whatever approach you've chosen.
|
||||
|
||||
To learn more, read the [full paper](https://arxiv.org/abs/2205.09712).
|
||||
|
||||
### Faithful reasoning architecture
|
||||
|
||||
A few months after publishing the selection-inference prompting technique, the authors extended the technique in a follow-up paper, with ideas for:
|
||||
|
||||
- figuring out when the selection-inference cycle should stop or continue
|
||||
- adding a value function to help search over multiple reasoning paths
|
||||
- reducing hallucination of fake facts by fine-tuning a model to reason about sentence labels (e.g., sen1) rather than writing out the sentences themselves
|
||||
|
||||
#### Method
|
||||
|
||||
In the original selection-inference technique, specialized 'selection' and 'inference' prompts are alternated to select facts and make inferences from those facts, combining to generate a sequence of reasoning steps.
|
||||
|
||||
The authors extend this technique with two additional components.
|
||||
|
||||
First, the authors add a 'halter' model that, after each inference step, is asked whether the inferences thus far are sufficient to answer the question. If yes, then the model generates a final answer.
|
||||
|
||||
The halter models brings a couple of advantages:
|
||||
|
||||
- it can tell the selection-inference process to stop or keep going, as necessary.
|
||||
- if the process never halts, you'll get no answer, which is often preferrable to a hallucinated guess
|
||||
|
||||
[
|
||||
<br>Source: *Faithful Reasoning Using Large Language Models* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
|
||||
|
||||
[
|
||||
<br>Source: *Faithful Reasoning Using Large Language Models* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
|
||||
|
||||
Second, the authors add a value function, which is used to assess the quality of reasoning steps and search over multiple reasoning trajectories. This echoes a common theme for increasing reliability; instead of generating a single answer from the model, generate a set of answers and then use some type of value function / discriminator / verifier model to pick the best one.
|
||||
|
||||
[
|
||||
<br>Source: *Faithful Reasoning Using Large Language Models* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
|
||||
|
||||
In addition to these two extensions, the authors also use a trick to reduce hallucination of fake facts. Rather than asking the model to write out factual sentences, they fine-tune a model to work with sentence labels (e.g., sen1) instead. This helps prevent the model from hallucinating fake facts not mentioned in the prompt context.
|
||||
|
||||
[
|
||||
<br>Source: *Faithful Reasoning Using Large Language Models* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
|
||||
|
||||
#### Results
|
||||
|
||||
The authors evaluated their technique on two benchmarks: the ProofWriter task (not shown) and [EntailmentBankQA](https://allenai.org/data/entailmentbank) (shown). The technique increased accuracy substantially, especially on harder reasoning problems.
|
||||
|
||||
|
||||
<br>Source: *Faithful Reasoning Using Large Language Models* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
|
||||
|
||||
In addition, their sentence label manipulation trick essentially eliminated hallucination!
|
||||
|
||||
|
||||
<br>Source: *Faithful Reasoning Using Large Language Models* by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
|
||||
|
||||
#### Implications
|
||||
|
||||
This paper illustrates a number of helpful lessons for improving the reliability of large language models:
|
||||
|
||||
- Split complex tasks into smaller, more reliable subtasks
|
||||
- Generate your answer in a step-by-step fashion, evaluating it along the way
|
||||
- Generate many possible answers and use another model or function to pick the ones that look best
|
||||
- Reduce hallucination by constraining what the model can say (e.g., by using sentence labels instead of sentences)
|
||||
- Maximize performance of models by fine-tuning them on specialized tasks
|
||||
|
||||
To learn more, read the [full paper](https://arxiv.org/abs/2205.09712).
|
||||
|
||||
### Least-to-most prompting
|
||||
|
||||
In addition to doing poorly on long reasoning chains (where selection-inference shines), chain-of-thought prompting can especially struggle when the examples are short but the task is long.
|
||||
|
||||
#### Method
|
||||
|
||||
Least-to-most prompting is another technique that splits up reasoning tasks into smaller, more reliable subtasks. The idea is to elicit a subtask from the model by prompting it with something like `To solve {question}, we need to first solve: "`. Then, with that subtask in hand, the model can generate a solution. The solution is appended to the original question and the process is repeated until a final answer is produced.
|
||||
|
||||
[
|
||||
<br>Source: *Least-to-most Prompting Enables Complex Reasoning in Large Language Models* by Denny Zhou et al. (2022)](https://arxiv.org/abs/2205.10625)
|
||||
|
||||
#### Results
|
||||
|
||||
When applied to benchmarks involving long reasoning chains using `code-davinci-002` (which is optimized for code but can still understand text), the authors measured gains as large as 16% -> 99.7%!
|
||||
|
||||
[
|
||||
|
||||
|
||||
|
||||
<br>Source: *Least-to-most Prompting Enables Complex Reasoning in Large Language Models* by Denny Zhou et al. (2022)](https://arxiv.org/abs/2205.10625)
|
||||
|
||||
#### Implications
|
||||
|
||||
Although the above gains from least-to-most prompting are impressive, they are measured on a very narrow set of tasks that require long reasoning chains.
|
||||
|
||||
Still, they illustrate a common theme: increase reliability by (a) breaking complex tasks into smaller subtasks and (b) giving the model more time and space to work out the answer.
|
||||
|
||||
To learn more, read the [full paper](https://arxiv.org/abs/2205.10625).
|
||||
|
||||
## Related ideas
|
||||
|
||||
### Maieutic prompting
|
||||
|
||||
#### Method
|
||||
|
||||
In contrast to the previous techniques, which try to maximize the likelihood of correct answers, another approach is to use GPT-3 to generate a tree of possible explanations (both correct *and incorrect*), and then analyze their relationships to guess at which set is correct. This technique was coined maieutic prompting by [Jaehun Jung et al. in May 2022](https://arxiv.org/abs/2205.11822) (maieutic means relating to the Socratic method of asking questions to elicit ideas).
|
||||
|
||||
The method is complicated, and works as follows:
|
||||
|
||||
- First, build a maieutic tree, where each node is a statement that could be true or false:
|
||||
- Start with a multiple-choice question or true/false statement (e.g. `War cannot have a tie`)
|
||||
- For each possible answer to the question, use the model to generate a correponding explanation (with a prompt like `War cannot have a tie? True, because`)
|
||||
- Then, prompt the model with the question and the generated explanation, and ask it to produce the answer. If reversing the explanation (with a prefix like `It is wrong to say that {explanation}`) reverses the answer, then the explanation is considered 'logically integral.'
|
||||
- If an explanation is not logically integral, then repeat the above process recursively, with each explanation turned into a True or False question, and generate more explanations for each new question.
|
||||
- After all of the recursive explaining is done, you end up with a tree of explanations, where each leaf on the tree has the property that reversing the explanation reverses the model's answer.
|
||||
- Second, convert the tree into a graph of relations:
|
||||
- For each node in the tree, calculate the model's relative belief in each node (inferred from the probability of getting an answer of `True` to given an explanation)
|
||||
- For each pair of nodes in the tree, use the model to identify whether they are entailed (implied) or contradicted
|
||||
- Third, find the most consistent set of beliefs and take those to be true:
|
||||
- Specifically, using the strength of belief in each node and the logical relationships between them, formulate the problem as a weighted maximum satisfiability problem (MAX-SAT)
|
||||
- Use a solver to the find the most self-consistent set of beliefs, and take those as true
|
||||
|
||||
[
|
||||
|
||||
|
||||
<br>Source: *Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations* by Jaehun Jung et al. (2022)](https://arxiv.org/abs/2205.11822)
|
||||
|
||||
|
||||
#### Results
|
||||
|
||||
[
|
||||
<br>Source: *Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations* by Jaehun Jung et al. (2022)](https://arxiv.org/abs/2205.11822)
|
||||
|
||||
#### Implications
|
||||
|
||||
Beyond the complexity, one limitation of this method is that it appears to only apply to questions that can be posed as multiple-choice.
|
||||
|
||||
To learn more, read the [full paper](https://arxiv.org/abs/2205.11822).
|
||||
|
||||
## Extensions
|
||||
|
||||
### Self-consistency
|
||||
|
||||
#### Method
|
||||
|
||||
For tasks with a discrete set of answers, one simple way to improve reliability is to sample multiple explanations & answers from the model (using a positive temperature) and then pick the final answer that appears most often.
|
||||
|
||||
[
|
||||
<br>Source: *Self-Consistency Improves Chain of Thought Reasoning in Language Models* by Xuezhi Wang et al. (2022)](https://arxiv.org/abs/2203.11171)
|
||||
|
||||
#### Results
|
||||
|
||||
This technique lifted accuracies by anywhere from 1 to 24 percentage points on a suite of math and reasoning benchmarks. (Plotted below are results from Google's LaMDA model; using Google's larger PaLM model, the baselines were higher but the gains were a bit smaller.)
|
||||
|
||||
[
|
||||
<br>Source: *Self-Consistency Improves Chain of Thought Reasoning in Language Models* by Xuezhi Wang et al. (2022)](https://arxiv.org/abs/2203.11171)
|
||||
|
||||
#### Implications
|
||||
|
||||
Although this technique is simple to implement, it can be costly. Generating a set of 10 answers will increase your costs by 10x.
|
||||
|
||||
Also, as with many of these techniques, it applies only to tasks with a limited set of answers. For open-ended tasks where each answer is unique (such as writing a poem), it's not obvious what it would mean to pick the most common answer.
|
||||
|
||||
Lastly, this technique ought to be most beneficial when there are multiple paths or phrasings to reach an answer; if there's only one path, then the technique may not help at all. An extreme example: If the task was to generate a single token answer, then taking the most common token from 100 generations would be no different than taking the token with the highest logprobs (which you can get with a single generation at temperature=0).
|
||||
|
||||
### Verifiers
|
||||
|
||||
Another key technique for improving task performance is to train a verifier or discriminator model to evaluate the outputs of the main generative model. If the discriminator rejects the output, then you can resample the generative model until you get an acceptable output. In many cases, it's easier to judge an answer than it is to create an answer, which helps explain the power of this method.
|
||||
|
||||
#### Method
|
||||
|
||||
In 2021, OpenAI researchers applied this technique to grade school math problems, using the following procedure:
|
||||
|
||||
- First, they fine-tuned a model on questions and solutions
|
||||
- For each problem in the training set, they generated 100 solutions
|
||||
- Each of those 100 solutions was automatically labeled as either correct or incorrect, based on whether the final answer was correct
|
||||
- Using those solutions, with some labeled correct and some labeled incorrect, they fine-tuned a verifier model to classify whether a question and candidate solution was correct or incorrect
|
||||
- Finally, at test time, the generative model creates 100 solutions to each problem, and the one with the highest score according to the verifier model is picked as the final answer
|
||||
|
||||
[
|
||||
<br>Source: *Training Verifiers to Solve Math Word Problems* by Karl Cobbe et al. (2021)](https://arxiv.org/abs/2110.14168)
|
||||
|
||||
#### Results
|
||||
|
||||
With a 175B GPT-3 model and 8,000 training examples, this technique substantially lifted gradeschool math accuracy from ~33% to ~55%.
|
||||
|
||||
[
|
||||
<br>Source: *Training Verifiers to Solve Math Word Problems* by Karl Cobbe et al. (2021)](https://arxiv.org/abs/2110.14168)
|
||||
|
||||
#### Implications
|
||||
|
||||
Similar to the self-consistency technique, this method can get expensive, as generating, say, 100 solutions per task will increase your costs by roughly ~100x.
|
||||
|
||||
## Theories of reliability
|
||||
|
||||
Although the techniques above vary in their approach, they all share the goal of improving reliability on complex tasks. Mainly they do this by:
|
||||
|
||||
- decomposing unreliable operations into smaller, more reliable operations (e.g., selection-inference prompting)
|
||||
- using multiple steps or multiple relationships to make the system's reliability greater than any individual component (e.g., maieutic prompting)
|
||||
|
||||
### Probabilistic graphical models
|
||||
|
||||
This paradigm of trying to build a reliable system out of less reliable components is reminiscent of probabilistic programming, and many of the analysis techniques of that field can be applied to this one.
|
||||
|
||||
In the paper *Language Model Cascades*, David Dohan et al. interpret the above techniques in the paradigm of probabilistic graphical models:
|
||||
|
||||
#### Chain of thought prompting
|
||||
|
||||
[
|
||||
<br>Source: *Language Model Cascades* by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
|
||||
|
||||
#### Fine-tuned chain of thought prompting / Self-taught reasoner
|
||||
|
||||
[
|
||||
<br>Source: *Language Model Cascades* by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
|
||||
|
||||
#### Selection-inference prompting
|
||||
|
||||
[
|
||||
<br>Source: *Language Model Cascades* by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
|
||||
|
||||
#### Verifiers
|
||||
|
||||
[
|
||||
<br>Source: *Language Model Cascades* by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
|
||||
|
||||
#### Implications
|
||||
|
||||
Although formulating these techniques as probabilistic graphical models may not be immediately useful for solving any particular problem, the framework may be helpful in selecting, combining, and discovering new techniques.
|
||||
|
||||
## Closing thoughts
|
||||
|
||||
Research into large language models is very active and evolving rapidly. Not only do researchers continue to improve the models, they are also continue to improve our understanding of how to best employ the models. To underscore the pace of these developments, note that all of the papers shared above were published within the past 12 months (as I write in Sep 2022).
|
||||
|
||||
In the future, expect better models and better techniques to be published. Even if the specific techniques here are eclipsed by future best practices, the general principles behind them will likely remain a key part of any expert user's toolkit.
|
||||
|
||||
## Bibliography
|
||||
|
||||
| Lesson | Paper | Date |
|
||||
|--------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|----------|
|
||||
| Break complex tasks into simpler subtasks (and consider exposing the intermediate outputs to users) | [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://arxiv.org/abs/2110.01691) | 2021 Oct |
|
||||
| You can improve output by generating many candidates, and then picking the one that looks best | [Training Verifiers to Solve Math Word Problems](https://arxiv.org/abs/2110.14168) | 2021 Oct |
|
||||
| On reasoning tasks, models do better when they reason step-by-step before answering | [Chain of Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) | 2022 Jan |
|
||||
| You can improve step-by-step reasoning by generating many explanation-answer outputs, and picking the most popular answer | [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171) | 2022 Mar |
|
||||
| If you want to fine-tune a step-by-step reasoner, you can do it with multiple-choice question & answer data alone | [STaR: Bootstrapping Reasoning With Reasoning](https://arxiv.org/abs/2203.14465) | 2022 Mar |
|
||||
| The step-by-step reasoning method works great even with zero examples | [Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916) | 2022 May |
|
||||
| You can do better than step-by-step reasoning by alternating a ‘selection’ prompt and an ‘inference’ prompt | [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://arxiv.org/abs/2205.09712) | 2022 May |
|
||||
| On long reasoning problems, you can improve step-by-step reasoning by splitting the problem into pieces to solve incrementally | [Least-to-most Prompting Enables Complex Reasoning in Large Language Models](https://arxiv.org/abs/2205.10625) | 2022 May |
|
||||
| You can have the model analyze both good and bogus explanations to figure out which set of explanations are most consistent | [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://arxiv.org/abs/2205.11822) | 2022 May |
|
||||
| You can think about these techniques in terms of probabilistic programming, where systems comprise unreliable components | [Language Model Cascades](https://arxiv.org/abs/2207.10342) | 2022 Jul |
|
||||
| You can eliminate hallucination with sentence label manipulation, and you can reduce wrong answers with a 'halter' prompt | [Faithful Reasoning Using Large Language Models](https://arxiv.org/abs/2208.14271) | 2022 Aug |
|
||||
@ -34,7 +34,7 @@ We've written guides and code examples for transitioning from the deprecated API
|
||||
[Guide: How to transition off the Classifications endpoint](https://help.openai.com/en/articles/6272941-classifications-transition-guide)
|
||||
|
||||
* Option 1: transition to fine-tuning **(recommended)**
|
||||
* Example code: [Fine-tuned_classification.ipynb](../examples/Fine-tuned_classification.ipynb)
|
||||
* Example code: [Classification.ipynb](../examples/Classification.ipynb)
|
||||
* Option 2: transition to embeddings
|
||||
* Example code: [Semantic_text_search_using_embeddings.ipynb](../examples/Semantic_text_search_using_embeddings.ipynb)
|
||||
* Option 3: reimplement Classifications endpoint functionality
|
||||
|
||||
@ -133,7 +133,7 @@ def classifications(
|
||||
{{ an optional instruction }}
|
||||
|
||||
Text: example 1 text
|
||||
Category: example 1 label
|
||||
Category: example 2 label
|
||||
---
|
||||
Text: example 1 text
|
||||
Category: example 2 label
|
||||
|
||||
@ -35,7 +35,7 @@ def get_score(context, query, log_probs, text_offsets) -> float:
|
||||
|
||||
def search(query, documents, engine):
|
||||
|
||||
prompts = [construct_context(query, doc) for doc in [""] + documents]
|
||||
prompts = [construct_context(query, doc) for doc in [""] + docs]
|
||||
|
||||
resps = openai.Completion.create(
|
||||
model=engine,
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user