Tisane is a natural language processing library, providing:
Tisane has monolithic architecture. All the functions are exposed using the same language models and the same analysis process invoked using the POST /parse method. Other methods in the API are either wrappers based on the process, helper methods, or allow inspection of the language models.
The current section of the documentation describes the two structures used in the parsing & transformation methods.
This guide describes how to setup your Tisane account. The steps you need to complete are as follows:
Navigate to Sign up to Tisane API. The free Community Plan allows up to 50,000 requests but comes with a limitation of 10 requests per minute.
You will need the API key to make requests. Open your Developer Profile to find your API keys.
In summary, the POST /parse method has 3 attributes: content, language, and settings. All 3 attributes are mandatory.
For example:{"language": "en", "content": "hello", "settings": {}}
Read on for more info on the response and the settings specs. The method doc pages contain snippets of code for your favorite languages and platforms.
The response of the POST /parse method contains several sections displayed or hidden according to the settings provided.
The common attributes are:
text
(string) - the original inputreduced_output
(boolean) - if the input is too big, and verbose information like the lexical chunk was requested, the verbose information will not be generated, and this flag will be set to true
and returned as part of the responsesentiment
(floating-point number) - a number in range -1 to 1 indicating the document-level sentiment. Only shown when document_sentiment
setting is set to true
.signal2noise
(floating-point number) - a signal to noise ranking of the text, in relation to the array of concepts specified in the relevant
setting. Only shown when the relevant
setting exists.The abuse
section is an array of detected instances of content that may violate some terms of use. NOTE: the terms of use in online communities may vary, and so it is up to the administrators to determine whether the content is indeed abusive. For instance, it makes no sense to restrict sexual advances in a dating community, or censor profanities when it’s accepted in the bulk of the community.
The section exists if instances of abuse are detected and the abuse
setting is either omitted or set to true
.
Every instance contains the following attributes:
offset
(unsigned integer) - zero-based offset where the instance startslength
(unsigned integer) - length of the contentsentence_index
(unsigned integer) - zero-based index of the sentence containing the instancetext
(string) - fragment of text containing the instance (only included if the snippets
setting is set to true
)tags
(array of strings) - when exists, provides additional detail about the abuse. For instance, if the fragment is classified as an attempt to sell hard drugs, one of the tags will be hard_drug.type
(string) - the type of the abuseseverity
(string) - how severe the abuse is. The levels of severity are low
, medium
, high
, and extreme
explanation
(string) - when available, provides rationale for the annotation; set the explain
setting to true
to enable.The currently supported types are:
personal_attack
- an insult / attack on the addressee, e.g. an instance of cyberbullying. Please note that an attack on a post or a point, or just negative sentiment is not the same as an insult. The line may be blurred at times. See our Knowledge Base for more information.bigotry
- hate speech aimed at one of the protected classes. The hate speech detected is not just racial slurs, but, generally, hostile statements aimed at the group as a wholeprofanity
- profane language, regardless of the intentsexual_advances
- welcome or unwelcome attempts to gain some sort of sexual favor or gratificationcriminal_activity
- attempts to sell or procure restricted items, criminal services, issuing death threats, and so onexternal_contact
- attempts to establish contact or payment via external means of communication, e.g. phone, email, instant messaging (may violate the rules in certain communities, e.g. gig economy portals, e-commerce portals)adult_only
- activities restricted for minors (e.g. consumption of alcohol)mental_issues
- content indicative of suicidal thoughts or depressionallegation
- claimed knowledge or accusation of a misconduct (not necessarily crime)provocation
- content likely to provoke an individual or a groupdisturbing
- graphic descriptions that may disturb readersno_meaningful_content
- unparseable gibberish without apparent meaningdata_leak
- private data like passwords, ID numbers, etc.spam
- (RESERVED) spam contentgeneric
- undefinedThe sentiment_expressions
section is an array of detected fragments indicating the attitude towards aspects or entities.
The section exists if sentiment is detected and the sentiment
setting is either omitted or set to true
.
Every instance contains the following attributes:
offset
(unsigned integer) - zero-based offset where the instance startslength
(unsigned integer) - length of the contentsentence_index
(unsigned integer) - zero-based index of the sentence containing the instancetext
(string) - fragment of text containing the instance (only included if the snippets
setting is set to true
)polarity
(string) - whether the attitude is positive
, negative
, or mixed
. Additionally, there is a default
sentiment used for cases when the entire snippet has been pre-classified. For instance, if a review is split into two portions, What did you like? and What did you not like?, and the reviewer replies briefly, e.g. The quiet. The service, the utterance itself has no sentiment value. When the calling application is aware of the intended sentiment, the default sentiment simply provides the targets / aspects, which will be then added the sentiment externally.targets
(array of strings) - when available, provides set of aspects and/or entities which are the targets of the sentiment. For instance, when the utterance is, The breakfast was yummy but the staff is unfriendly, the targets for the two sentiment expressions are meal
and staff
. Named entities may also be targets of the sentiment.reasons
(array of strings) - when available, provides reasons for the sentiment. In the example utterance above (The breakfast was yummy but the staff is unfriendly), the reasons
array for the staff
is ["unfriendly"]
, while the reasons
array for meal
is ["tasty"]
.explanation
(string) - when available, provides rationale for the sentiment; set the explain
setting to true
to enable.Example:
"sentiment_expressions": [
{
"sentence_index": 0,
"offset": 0,
"length": 32,
"polarity": "positive",
"reasons": ["close"],
"targets": ["location"]
},
{
"sentence_index": 0,
"offset": 38,
"length": 29,
"polarity": "negative",
"reasons": ["disrespectful"],
"targets": ["staff"]
}
]
The entities_summary
section is an array of named entity objects detected in the text.
The section exists if named entities are detected and the entities
setting is either omitted or set to true
.
Every entity contains the following attributes:
name
(string) - the most complete name of the entity in the text of all the mentionsref_lemma
(string) - when available, the dictionary form of the entity in the reference language (English) regardless of the input languagetype
(string) - a string or an array of strings specifying the type of the entity, such as person
, organization
, numeric
, amount_of_money
, place
. Certain entities, like countries, may have several types (because a country is both a place
and an organization
).subtype
(string) - a string indicating the subtype of the entitymentions
(array of objects) - a set of instances where the entity was mentioned in the textEvery mention contains the following attributes:
offset
(unsigned integer) - zero-based offset where the instance startslength
(unsigned integer) - length of the contentsentence_index
(unsigned integer) - zero-based index of the sentence containing the instancetext
(string) - fragment of text containing the instance (only included if the snippets
setting is set to true
)Example:
"entities_summary": [
{
"type": "person",
"name": "John Smith",
"ref_lemma": "John Smith",
"mentions": [
{
"sentence_index": 0,
"offset": 0,
"length": 10 }
]
}
,
{
"type": [ "organization", "place" ]
,
"name": "UK",
"ref_lemma": "U.K.",
"mentions": [
{
"sentence_index": 0,
"offset": 40,
"length": 2 }
]
}
]
The currently supported entity types are:
person
, with optional subtypes: fictional_character
, important_person
, spiritual_being
organization
(note that a country is both an organization and a place)place
time_range
date
time
hashtag
email
amount_of_money
phone
phone number, either domestic or international, in a variety of formatsrole
(a social role, e.g. position in an organization)software
website
(URL), with an optional subtype: tor
for Onion links; note that web services may also have the software
type assignedweight
bank_account
only IBAN format is supported; subtypes: iban
credit_card
, with optional subtypes: visa
, mastercard
, american_express
, diners_club
, discovery
, jcb
, unionpay
coordinates
(GPS coordinates)credential
, with optional subtypes: md5
, sha-1
crypto
, with optional subtypes: bitcoin
, ethereum
, monero
, monero_payment_id
, litecoin
, dash
event
file
only Windows pathnames are supported; subtypes: windows
, facebook
(for images downloaded from Facebook)flight_code
identifier
ip_address
, subtypes: v4
, v6
mac_address
numeric
(an unclassified numeric entity)username
The topics
section is an array of topics (subjects, domains, themes in other terms) detected in the text.
The section exists if topics are detected and the topics
setting is either omitted or set to true
.
By default, a topic is a string. If topic_stats
setting is set to true
, then every entry in the array contains:
topic
(string) - the topic itselfcoverage
(floating-point number) - a number between 0 and 1, indicating the ratio between the number of sentences where the topic is detected to the total number of sentencesThe memory
section contains optional context to pass to the settings
in subsequent messages in the same conversation thread. See Context and Long-Term Memory for more details.
Tisane allows obtaining more in-depth data, specifically:
The sentence_list
section is generated if the words
or the parses
setting is set to true
.
Every sentence structure in the list contains:
offset
(unsigned integer) - zero-based offset where the sentence startslength
(unsigned integer) - length of the sentencetext
(string) - the sentence itselfcorrected_text
(string) - if a misspelling was detected and the spellchecking is active, contains the automatically corrected textwords
(array of structures) - if words
setting is set to true
, generates extended information about every lexical chunk. (The term “word” is used for the sake of simplicity, however, it may not be linguistically correct to equate lexical chunks with words.)parse_tree
(object) - if parses
setting is set to true
, generates information about the parse tree and the phrases detected in the sentence.nbest_parses
(array of parse objects) if parses
setting is set to true
and deterministic
setting is set to false
, generates information about the parse trees that were deemed close enough to the best one but not the best.Every lexical chunk (“word”) structure in the words
array contains:
type
(string) - the type of the element: punctuation
for punctuation marks, numeral
for numerals, or word
for everything elsetext
(string) - the textoffset
(unsigned integer) - zero-based offset where the element startslength
(unsigned integer) - length of the elementcorrected_text
(string) - if a misspelling is detected, the corrected formlettercase
(string) - the original letter case: upper
, capitalized
, or mixed
. If lowercase or no case, the attribute is omitted.stopword
(boolean) - determines whether the word is a stopwordgrammar
(array of strings or structures) - generates the list of grammar features associated with the word
. If the feature_standard
setting is defined as native
, then every feature is an object containing a numeral (index
) and a string (value
). Otherwise, every feature is a plain stringFor lexical words only:
role
(string) - semantic role, like agent
or patient
. Note that in passive voice, the semantic roles are reverse to the syntactic roles. E.g. in a sentence like The car was driven by David, car is the patient, and David is the agent.numeric_value
(floating-point number) - the numeric value, if the chunk has a value associated with itfamily
(integer number) - the ID of the family associated with the disambiguated word-sense of the lexical chunkdefinition
(string) - the definition of the family, if the fetch_definitions
setting is set to true
lexeme
(integer number) - the ID of the lexeme entry associated with the disambiguated word-sense of the lexical chunknondictionary_pattern
(integer number) - the ID of a non-dictionary pattern that matched, if the word was not in the language model but was classified by the nondictionary heuristicsstyle
(array of strings or structures) - generates the list of style features associated with the word
. Only if the feature_standard
setting is set to native
or description
semantics
(array of strings or structures) - generates the list of semantic features associated with the word
. Only if the feature_standard
setting is set to native
or description
segmentation
(structure) - generates info about the selected segmentation, if there are several possibilities to segment the current lexical chunk and the deterministic
setting is set to false
. A segmentation is simply an array of word
structures.other_segmentations
(array of structures) - generates info about the segmentations deemed incorrect during the disambiguation process. Every entry has the same structure as the segmentation
structure.nbest_senses
(array of structures) - when the deterministic
setting is set to false
, generates a set of hypotheses that were deemed incorrect by the disambiguation process. Every hypothesis contains the following attributes: grammar
, style
, and semantics
, identical in structure to their counterparts above; and senses
, an array of word-senses associated with every hypothesis. Every sense has a family
, which is an ID of the associated family; and, if the fetch_definitions
setting is set to true
, definition
and ref_lemma
of that family.For punctuation marks only:
id
(integer number) - the ID of the punctuation markbehavior
(string) - the behavior code of the punctuation mark. Values: sentenceTerminator
, genericComma
, bracketStart
, bracketEnd
, scopeDelimiter
, hyphen
, quoteStart
, quoteEnd
, listComma
(for East-Asian enumeration commas like 、)Every parse tree, or more accurately, parse forest, is a collection of phrases, hierarchically linked to each other.
At the top level of the parse, there is an array of root phrases under the phrases
element and the numeric id
associated with it. Every phrase may have children phrases. Every phrase has the following attributes:
type
(string) - a Penn treebank phrase tag denoting the type of the phrase, e.g. S, VP, NP, etc.family
(integer number) - an ID of the phrase familyoffset
(unsigned integer) - a zero-based offset where the phrase startslength
(unsigned integer) - the span of the phraserole
(string) - the semantic role of the phrase, if any, analogous to that of the wordstext
(string) - the phrase text, where the phrase members are delimited by the vertical bar character. Children phrases are enclosed in brackets. E.g., driven|by|David or (The|car)|was|(driven|by|David).Example:
"parse_tree": {
"id": 4,
"phrases": [
{
"type": "S",
"family": 1451,
"offset": 0,
"length": 27,
"text": "(The|car)|was|(driven|by|David)",
"children": [
{
"type": "NP",
"family": 1081,
"offset": 0,
"length": 7,
"text": "The|car",
"role": "patient"
},
{
"type": "VP",
"family": 1172,
"offset": 12,
"length": 15,
"text": "driven|by|David",
"role": "verb"
}
]
}
Tisane supports automatic, context-aware spelling correction. Whether it’s a misspelling or a purported obfuscation, Tisane attempts to deduce the intended meaning, if the language model does not recognize the word.
When or if it’s found, Tisane adds the corrected_text
attribute to the word (if the words / lexical chunks are returned) and the sentence (if the sentence text is generated). Sentence-level corrected_text
is displayed if words
or parses
are set to true.
Note that as Tisane works with large dictionaries, you may need to exclude more esoteric terms by using the min_generic_frequency
setting.
Note that the invocation of spell-checking does not depend on whether the sentences and the words sections are generated in the output. The spellchecking can be disabled by setting disable_spellcheck
to true
. Another option is to enable the spellchecking for lowercase words only, thus excluding potential proper nouns in languages that support capitalization; to avoid spell-checking capitalized and uppercase words, set lowercase_spellcheck_only
to true
.
The purpose of the settings structure is to:
All settings are optional. To leave all settings to default, simply provide an empty object ({}
).
format
(string) - the format of the content. Some policies will be applied depending on the format. Certain logic in the underlying language models may require the content to be of a certain format (e.g. logic applied on the reviews may seek for sentiment more aggressively). The default format is empty / undefined. The format values are:
review
- a review of a product or a service or any other review. Normally, the underlying language models will seek for sentiment expressions more aggressively in reviews.dialogue
- a comment or a post which is a part of a dialogue. An example of a logic more specific to a dialogue is name calling. A single word like “idiot” would not be a personal attack in any other format, but it is certainly a personal attack when part of a dialogue.shortpost
- a microblogging post, e.g. a tweet.longform
- a long post or an article.proofread
- a post which was proofread. In the proofread posts, the spellchecking is switched off.alias
- a nickname in an online community.search
- a search query. Search queries may not always be grammatically correct. Certain topics and items, that we may otherwise let pass, are tagged with the search
format.disable_spellcheck
(boolean) - determines whether the automatic spellchecking is to be disabled. Default: false
.
lowercase_spellcheck_only
(boolean) - determines whether the automatic spellchecking is only to be applied to words in lowercase. Default: false
min_generic_frequency
(int) - allows excluding more esoteric terms; the valid values are 0 thru 10.
subscope
(boolean) - enables sub-scope parsing, for scenarios like hashtag, URL parsing, and obfuscated content (e.g. ihateyou). Default: false
.
lang_detect_segmentation_regex
(string) - allows defining custom language detection fragment boundaries. For example, if multiple languages may be used in different sentences in the same text, you may want to define the regex as: (([\r\n]|[.!?][ ]))
.
domain_factors
(set of pairs made of strings and numbers) - provides a session-scope cues for the domains of discourse. This is a powerful tool that allows tailoring the result based on the use case. The format is, family ID of the domain as a key and the multiplication factor as a value (e.g. “12345”: 5.0). For example, when processing text looking for criminal activity, we may want to set domains relevant to drugs, firearms, crime, higher: "domain_factors": {"31058": 5.0, "45220": 5.0, "14112": 5.0, "14509": 3.0, "28309": 5.0, "43220": 5.0, "34581": 5.0}
. The same device can be used to eliminate noise coming from domains we know are irrelevant by setting the factor to a value lower than 1.
when
(date string, format YYYY-MM-DD) - indicates when the utterance was uttered. (TO BE IMPLEMENTED) The purpose is to prune word senses that were not available at a particular point in time. For example, the words troll, mail, and post had nothing to do with the Internet 300 years ago because there was no Internet, and so in a text that was written hundreds of years ago, we should ignore the word senses that emerged only recently.
abuse
(boolean) - output instances of abusive content (default: true
)
sentiment
(boolean) - output sentiment-bearing snippets (default: true
)
document_sentiment
(boolean) - output document-level sentiment (default: false
)
entities
(boolean) - output entities (default: true
)
topics
(boolean) - output topics (default: true
), with two more relevant settings:
topic_stats
(boolean) - include coverage statistics in the topic output (default: false
). When set, the topic is an object containing the attributes topic
(string) and coverage
(floating-point number). The coverage indicates a share of sentences touching the topic among all the sentences.optimize_topics
(boolean) - if true
, the less specific topics are removed if they are parts of the more specific topics. For example, when the topic is cryptocurrency
, the optimization removes finance
.words
(boolean) - output the lexical chunks / words for every sentence (default: false
). In languages without white spaces (Chinese, Japanese, Thai), the tokens are tokenized words. In languages with compounds (e.g. German, Dutch, Norwegian), the compounds are split.
fetch_definitions
(boolean) - include definitions of the words in the output (default: false
). Only relevant when the words
setting is true
parses
(boolean) - output parse forests of phrases
deterministic
(boolean) - whether the n-best senses and n-best parses are to be output in addition to the detected sense. If true
, only the detected sense will be output. Default: true
snippets
(boolean) - include the text snippets in the abuse, sentiment, and entities sections (default: false
)
explain
(boolean) - if true
, a reasoning for the abuse and sentiment snippets is provided when possible (see the explanation
attribute)
feature_standard
(string) - determines the standard used to output the features (grammar, style, semantics) in the response object. The standards we support are:
ud
: Universal Dependencies tags (default)penn
: Penn treebank tagsnative
: Tisane native feature codesdescription
: Tisane native feature descriptionsOnly the native Tisane standards (codes and descriptions) support style and semantic features.
topic_standard
(string) - determines the standard used to output the topics in the response object. The standards we support are:
iptc_code
- IPTC topic taxonomy codeiptc_description
- IPTC topic taxonomy descriptioniab_code
- IAB topic taxonomy codeiab_description
- IAB topic taxonomy descriptionnative
- Tisane domain description, coming from the family description (default)sentiment_analysis_type
(string) - the type of the sentiment analysis strategy. The values are:
products_and_services
- most common sentiment analysis of products and servicesentity
- sentiment analysis with entities as targetscreative_content_review
- reviews of creative content (RESERVED)political_essay
- political essays (RESERVED)Human understanding of language is not a simple “sliding window” with scope limited to a sentence. Language is accompanied by gestures, visuals, and knowledge of the previous communication. Sometimes, code-words may be used to conceal the words’ original meaning.
When detecting abuse, a name of an ethnicity or a religious group may be not offensive, but when superimposed over a picture of an ape or a pig, it is meant of offend. When translating from a language without gender distinctions in verbs (like English) to a language with distinctions (like Russian or Hebrew), there is no way to know from an utterance alone if the speaker is female. When a scammer is collecting details piecemeal over a series of utterances, knowledge of previous utterances is needed to take action.
Tisane’s Memory module allows pre-initializing the analysis, as well as reassigning meanings, and more. The module is made of three simple components that are flexible enough for a variety of tasks:
Reassignments define the attributes to set based on other attributes. This allows to:
and more, within a scope of a call.
The assign
section is an array of structures defining:
if
- conditions to match:regex
- a regular expression (RE2 syntax)family
- a family IDfeatures
- a list of feature values. A feature is a structure with an index
and a value
. For example: {"index":1, "value":"NOUN"}
.hypernym
- a family ID of a hypernymthen
- attributes to assignfamily
- a family IDfeatures
- a list of feature values. A feature is a structure with an index
and a value
. For example: {"index":1, "value":"NOUN"}
.hypernym
- a family ID of a hypernymExamples:
An array of flag structures that add some context. A flag is a structure with an index
and a value
. For example: {"index":36, "value":"WFH"}
.
Aside from the flags returned in the memory
section of the response, these flags can be set:
{"index":36, "value":"PEBD"}
(agents_of_bad_things) - the context is about a bad player or an agent responsible for bad things{"index":36, "value":"BADANML"}
(bad_animal) - the context is an animal that symbolizes bad qualities (e.g. pig, ape, snake, etc.){"index":36, "value":"BULKMSG"}
(bulk_message) - the message was sent in bulk{"index":36, "value":"DETHR"}
(death_related) - the context is something related to death{"index":36, "value":"EARNMUCH"}
(make_money) - the context is related to making money{"index":36, "value":"IDEP"}
(my_departure) - the author of the text mentioned departing{"index":36, "value":"SECO"}
(sexually_conservative) - any attempt to exchange photos or anything that may be either sexual or non-sexual is to be deemed sexual{"index":36, "value":"TRPA"}
(trusted_party) - the author of the text claims to be a trusted party (e.g. a relative or a spouse){"index":36, "value":"WSTE"}
(waste) - the context is about waste, organic or inorganic{"index":36, "value":"WOPR"}
(won_prize) - prize or money winning was mentioned or implied{"index":36, "value":"WFH"}
(work_from_home) - work from home was mentioned{"index":5, "value":"ORG"}
(organization) - an organization was mentioned{"index":5, "value":"ROLE"}
(role) - a role or a position was mentionedThe section contains structures to be used in coreference resolution. The attributes are:
family
- the family ID of the antecedentfeatures
- the list of features. Every feature is a structure with an index
and a value
. For example: {"index":36, "value":"WFH"}
.When we’re studying a bunch of posts commenting on an issue or an article, we may want to prioritize the ones more relevant to the topic, and containing more reason and logic than emotion. This is what the signal to noise ranking is meant to achieve.
The signal to noise ranking is made of two parts:
To determine the most relevant concepts, we need to analyze the headline or the article itself. The headline is usually enough. We need two additional settings:
keyword_features
(an object of strings with string values) - determines the features to look for in a word. When such a feature is found, the family ID is added to the set of potentially relevant family IDs.stop_hypernyms
(an array of integers) - if a potentially relevant family ID has a hypernym listed in this setting, it will not be considered. For example, we extracted a set of nouns from the headline, but we may not be interested in abstractions or feelings. E.g. from a headline like Fear and Loathing in Las Vegas we want Las Vegas only. Optional.If keyword_features
is provided in the settings, the response will have a special attribute, relevant
, containing a set of family IDs.
At the second stage, when ranking the actual posts or comments for relevance, this array is to be supplied among the settings. The ranking is boosted when the domain, the hypernyms, or the families related to those in the relevant
array are mentioned, when negative and positive sentiment is linked to aspects, and penalized when the negativity is not linked to aspects, or abuse of any kind is found. The latter consideration may be disabled, e.g. when we are looking for specific criminal content. When the abuse_not_noise
parameter is specified and set to true
, the abuse is not penalized by the ranking calculations.
To sum it up, in order to calculate the signal to noise ranking:
keyword_features
and, optionally, stop_hypernyms
in the settings. Obtain the relevant
attribute.relevant
attribute obtained in step 1.Use this page to mock Tisane API Documentation in your testing and development.
Run our mock API sample using the open source WireMock library, or in the free edition of WireMock Cloud. You'll have a working API server simulating the behavior of Tisane API Documentation, which will allow you to keep building and testing even if the actual API you isn't currently available.