blob: 43d1c84a7094e4c8726ec34e4591c67e96affee1 [file] [log] [blame]
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="keywords" content="installation, operating, docker, docker-compose, devops, logging, logstash, elk, monitoring, prometheus, grafana">
<title> Operating Ditto • Eclipse Ditto • a digital twin framework</title>
<link rel="stylesheet" href="css/syntax.css">
<link rel="stylesheet" type="text/css" href="//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css" crossorigin="anonymous">
<link rel="stylesheet" href="css/modern-business.css">
<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/css/bootstrap.min.css" crossorigin="anonymous">
<link rel="stylesheet" href="css/customstyles.css">
<link rel="stylesheet" href="css/boxshadowproperties.css">
<link rel="stylesheet" href="css/theme-ditto.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,700|Source+Code+Pro:300,600|Titillium+Web:400,600,700">
<script src="//cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js" crossorigin="anonymous"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/js/bootstrap.min.js" crossorigin="anonymous"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/anchor-js/2.0.0/anchor.min.js" crossorigin="anonymous"></script>
<script src="js/toc.js"></script>
<script src="js/customscripts.js"></script>
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "Organization",
"url": "https://eclipse.org/ditto/",
"logo": "https://eclipse.org/ditto/images/ditto.svg"
}
</script>
<link rel="icon" type="image/png" href="images/favicon-16x16.png" sizes="16x16">
<link rel="icon" type="image/png" href="images/favicon-32x32.png" sizes="32x32">
<link rel="icon" type="image/png" href="images/favicon-96x96.png" sizes="96x96">
<link rel="alternate" type="application/rss+xml" title="Eclipse Ditto Blog" href="https://www.eclipse.org/ditto/feed.xml">
<!-- Eclipse Foundation cookie consent: -->
<link rel="stylesheet" type="text/css" href="//www.eclipse.org/eclipse.org-common/themes/solstice/public/stylesheets/vendor/cookieconsent/cookieconsent.min.css" />
<script src="//www.eclipse.org/eclipse.org-common/themes/solstice/public/javascript/vendor/cookieconsent/default.min.js"></script>
<script>
$(document).ready(function() {
$("#tg-sb-link").click(function() {
$("#tg-sb-sidebar").toggle();
$("#tg-sb-content").toggleClass('col-md-9');
$("#tg-sb-content").toggleClass('col-md-12');
$("#tg-sb-icon").toggleClass('fa-toggle-on');
$("#tg-sb-icon").toggleClass('fa-toggle-off');
});
});
</script>
</head>
<script>
(function(w,d,s,l,i){
w[l]=w[l]||[];
w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});
var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),
dl=l!='dataLayer'?'&l='+l:'';
j.async=true;
j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;
f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-5WLCZXC');
</script>
<body>
<!-- Navigation -->
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container topnavlinks">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-ditto-home" href="index.html">&nbsp;<img src="images/ditto_allwhite_symbolonly.svg" class="ditto-navbar-symbol" alt="Home"> <img src="images/ditto_allwhite_textonly.svg" class="ditto-navbar-symbol-text" alt="Ditto"></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
<!-- toggle sidebar button -->
<!--<li><a id="tg-sb-link" href="#"><i id="tg-sb-icon" class="fa fa-toggle-on"></i> Nav</a></li>-->
<!-- entries without drop-downs appear here -->
<li><a href="blog.html">Blog</a></li>
<li><a href="intro-overview.html">Documentation</a></li>
<li><a href="http-api-doc.html">HTTP API</a></li>
<li><a href="sandbox.html">Sandbox</a></li>
<li><a href="https://github.com/eclipse/ditto" target="_blank">GitHub</a></li>
<li><a href="https://github.com/eclipse/ditto-examples" target="_blank">GitHub examples</a></li>
<!-- entries with drop-downs appear here -->
<!-- conditional logic to control which topnav appears for the audience defined in the configuration file.-->
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Links<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="https://projects.eclipse.org/projects/iot.ditto" target="_blank">Eclipse Ditto Project</a></li>
<li><a href="https://www.eclipse.org/forums/index.php/f/364/" target="_blank">Forum</a></li>
<li><a href="https://ci.eclipse.org/ditto/" target="_blank">Jenkins</a></li>
<li><a href="https://dev.eclipse.org/mhonarc/lists/ditto-dev/" target="_blank">Mailing list archives</a></li>
<li><a href="https://gitter.im/eclipse/ditto" target="_blank">Gitter.im chat</a></li>
</ul>
</li>
<!--comment out this block if you want to hide search-->
<li>
<!--start search-->
<div id="search-demo-container">
<input type="text" id="search-input" placeholder="search...">
<ul id="results-container"></ul>
</div>
<script src="//cdnjs.cloudflare.com/ajax/libs/simple-jekyll-search/0.0.9/jekyll-search.js" type="text/javascript"></script>
<script type="text/javascript">
SimpleJekyllSearch.init({
searchInput: document.getElementById('search-input'),
resultsContainer: document.getElementById('results-container'),
dataSource: 'search.json',
searchResultTemplate: '<li><a href="{url}" title="Operating Ditto">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
})
</script>
<!--end search-->
</li>
</ul>
</div>
</div>
<!-- /.container -->
</nav>
<!-- Page Content -->
<div class="container">
<div id="main">
<!-- Content Row -->
<div class="row">
<!-- Sidebar Column -->
<div class="col-md-3" id="tg-sb-sidebar">
<ul id="mysidebar" class="nav">
<li class="sidebarTitle">
<label for="docVersion">Eclipse Ditto version:</label>
<div class="select-wrapper">
<select id="docVersion" name="docVersion">
<option value="">development</option>
<option value="1.0">1.0</option>
<option value="1.1">1.1</option>
<option value="1.2">1.2</option>
<option value="1.3">1.3</option>
<option value="1.4">1.4</option>
</select>
</div>
<div id="dev-warning">
<div markdown="span" class="alert alert-warning" role="alert" style="font-size:0.6em"><i class="fa fa-warning"></i> <b>Important:</b> This documentation reflects the latest 'development'. You might want to choose a released version.</div>
</div>
</li>
<li class="subfolders">
<a href="#"><span></span>Introduction</a>
<ul>
<li><a href="intro-overview.html">Overview</a></li>
<li><a href="intro-digitaltwins.html">Digital twins</a></li>
<li><a href="intro-hello-world.html">Hello world</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>Release Notes</a>
<ul>
<li><a href="release_notes_130.html">1.3.0</a></li>
<li><a href="release_notes_121.html">1.2.1</a></li>
<li><a href="release_notes_120.html">1.2.0</a></li>
<li><a href="release_notes_115.html">1.1.5</a></li>
<li><a href="release_notes_113.html">1.1.3</a></li>
<li><a href="release_notes_112.html">1.1.2</a></li>
<li><a href="release_notes_111.html">1.1.1</a></li>
<li><a href="release_notes_110.html">1.1.0</a></li>
<li><a href="release_notes_100.html">1.0.0</a></li>
<li><a href="release_notes_090.html">0.9.0</a></li>
<li><a href="release_notes_080.html">0.8.0</a></li>
<li class="subfolders">
<a href="#"><span></span>Milestone releases</a>
<ul>
<li><a href="release_notes_100-M2.html">1.0.0-M2</a></li>
<li><a href="release_notes_100-M1a.html">1.0.0-M1a</a></li>
<li><a href="release_notes_090-M2.html">0.9.0-M2</a></li>
<li><a href="release_notes_090-M1.html">0.9.0-M1</a></li>
<li><a href="release_notes_080-M3.html">0.8.0-M3</a></li>
<li><a href="release_notes_080-M2.html">0.8.0-M2</a></li>
<li><a href="release_notes_080-M1.html">0.8.0-M1</a></li>
<li><a href="release_notes_030-M2.html">0.3.0-M2</a></li>
<li><a href="release_notes_030-M1.html">0.3.0-M1</a></li>
<li><a href="release_notes_020-M1.html">0.2.0-M1</a></li>
<li><a href="release_notes_010-M3.html">0.1.0-M3</a></li>
<li><a href="release_notes_010-M1.html">0.1.0-M1</a></li>
</ul>
</li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>Installation</a>
<ul>
<li><a href="installation-building.html">Building Ditto</a></li>
<li><a href="installation-running.html">Running Ditto</a></li>
<li class="active"><a href="installation-operating.html">Operating Ditto</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>Basic concepts</a>
<ul>
<li><a href="basic-overview.html">Overview</a></li>
<li class="subfolders">
<a href="#"><span></span>Model entities</a>
<ul>
<li><a href="basic-thing.html">Thing</a></li>
<li><a href="basic-acl.html">Access Control List (ACL)</a></li>
<li><a href="basic-feature.html">Feature</a></li>
<li><a href="basic-policy.html">Policy</a></li>
<li><a href="basic-namespaces-and-names.html">Namespaces and Names</a></li>
<li><a href="basic-metadata.html">Thing Metadata</a></li>
<li><a href="basic-errors.html">Errors</a></li>
</ul>
</li>
<li><a href="basic-auth.html">Authentication and Authorization</a></li>
<li><a href="basic-messages.html">Messages</a></li>
<li><a href="basic-signals.html">Signals</a></li>
<li class="subfolders">
<a href="#"><span></span>Signal types</a>
<ul>
<li><a href="basic-signals-command.html">Command</a></li>
<li><a href="basic-signals-commandresponse.html">Command response</a></li>
<li><a href="basic-signals-errorresponse.html">Error response</a></li>
<li><a href="basic-signals-event.html">Event</a></li>
</ul>
</li>
<li><a href="basic-apis.html">APIs</a></li>
<li><a href="basic-connections.html">Connections</a></li>
<li><a href="basic-placeholders.html">Placeholders</a></li>
<li><a href="basic-changenotifications.html">Change notifications</a></li>
<li><a href="basic-rql.html">RQL expressions</a></li>
<li><a href="basic-enrichment.html">Signal enrichment</a></li>
<li><a href="basic-search.html">Search</a></li>
<li><a href="basic-acknowledgements.html">Acknowledgements / QoS</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>Architecture</a>
<ul>
<li><a href="architecture-overview.html">Overview</a></li>
<li class="subfolders">
<a href="#"><span></span>Services</a>
<ul>
<li><a href="architecture-services-policies.html">Policies</a></li>
<li><a href="architecture-services-things.html">Things</a></li>
<li><a href="architecture-services-things-search.html">Things-Search</a></li>
<li><a href="architecture-services-connectivity.html">Connectivity</a></li>
<li><a href="architecture-services-concierge.html">Concierge</a></li>
<li><a href="architecture-services-gateway.html">Gateway</a></li>
</ul>
</li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>HTTP API</a>
<ul>
<li><a href="httpapi-overview.html">Overview</a></li>
<li><a href="httpapi-concepts.html">Concepts</a></li>
<li><a href="httpapi-search.html">Search</a></li>
<li><a href="httpapi-messages.html">Messages</a></li>
<li><a href="httpapi-protocol-bindings-websocket.html">WebSocket protocol binding</a></li>
<li><a href="httpapi-sse.html">Server sent events</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>Connectivity API</a>
<ul>
<li><a href="connectivity-overview.html">Overview</a></li>
<li><a href="connectivity-manage-connections.html">Manage connections</a></li>
<li><a href="connectivity-protocol-bindings-amqp091.html">AMQP 0.9.1 protocol binding</a></li>
<li><a href="connectivity-protocol-bindings-amqp10.html">AMQP 1.0 protocol binding</a></li>
<li><a href="connectivity-protocol-bindings-mqtt.html">MQTT 3.1.1 protocol binding</a></li>
<li><a href="connectivity-protocol-bindings-mqtt5.html">MQTT 5 protocol binding</a></li>
<li><a href="connectivity-protocol-bindings-http.html">HTTP 1.1 protocol binding</a></li>
<li><a href="connectivity-protocol-bindings-kafka2.html">Kafka 2.x protocol binding</a></li>
<li><a href="connectivity-mapping.html">Payload mapping</a></li>
<li><a href="connectivity-header-mapping.html">Header mapping</a></li>
<li><a href="connectivity-tls-certificates.html">TLS certificates</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>Client SDK</a>
<ul>
<li><a href="client-sdk-overview.html">Overview</a></li>
<li><a href="client-sdk-java.html">Java</a></li>
<li><a href="client-sdk-javascript.html">JavaScript</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>Ditto Protocol</a>
<ul>
<li><a href="protocol-overview.html">Overview</a></li>
<li><a href="protocol-twinlive.html">Twin/live channel</a></li>
<li><a href="protocol-specification.html">Specification</a></li>
<li><a href="protocol-specification-topic.html">Protocol topic</a></li>
<li><a href="protocol-specification-things.html">Things group</a></li>
<li class="subfolders">
<a href="#"><span></span>→ commands/events</a>
<ul>
<li><a href="protocol-specification-things-create-or-modify.html">Create/Modify</a></li>
<li><a href="protocol-specification-things-retrieve.html">Retrieve</a></li>
<li><a href="protocol-specification-things-delete.html">Delete</a></li>
<li><a href="protocol-specification-acks.html">Acknowledgements</a></li>
<li><a href="protocol-specification-errors.html">Errors</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>→ search/messages</a>
<ul>
<li><a href="protocol-specification-things-search.html">Search</a></li>
<li><a href="protocol-specification-things-messages.html">Messages</a></li>
</ul>
</li>
<li><a href="protocol-specification-policies.html">Policies group</a></li>
<li class="subfolders">
<a href="#"><span></span>→ commands/events</a>
<ul>
<li><a href="protocol-specification-policies-create-or-modify.html">Create/Modify</a></li>
<li><a href="protocol-specification-policies-retrieve.html">Retrieve</a></li>
<li><a href="protocol-specification-policies-delete.html">Delete</a></li>
</ul>
</li>
<li><a href="protocol-bindings.html">Bindings</a></li>
<li><a href="protocol-examples.html">Examples</a></li>
<li class="subfolders">
<a href="#"><span></span>→ Things examples</a>
<ul>
<li><a href="protocol-examples-creatething.html">Create a Thing</a></li>
<li><a href="protocol-examples-deletething.html">Delete a Thing</a></li>
<li><a href="protocol-examples-modifything.html">Modify a Thing</a></li>
<li><a href="protocol-examples-retrievething.html">Retrieve a Thing</a></li>
<li><a href="protocol-examples-retrievethings.html">Retrieve multiple Things</a></li>
<li><a href="protocol-examples-modifypolicyid.html">Modify the Policy ID of a Thing</a></li>
<li><a href="protocol-examples-createattributes.html">Create Attributes</a></li>
<li><a href="protocol-examples-deleteattributes.html">Delete Attributes</a></li>
<li><a href="protocol-examples-modifyattributes.html">Modify Attributes</a></li>
<li><a href="protocol-examples-retrieveattributes.html">Retrieve Attributes</a></li>
<li><a href="protocol-examples-createattribute.html">Create a single Attribute</a></li>
<li><a href="protocol-examples-deleteattribute.html">Delete a single Attribute</a></li>
<li><a href="protocol-examples-modifyattribute.html">Modify a single Attribute</a></li>
<li><a href="protocol-examples-retrieveattribute.html">Retrieve a single Attribute</a></li>
<li><a href="protocol-examples-createthingdefinition.html">Create a Definition</a></li>
<li><a href="protocol-examples-deletethingdefinition.html">Delete a Definition</a></li>
<li><a href="protocol-examples-modifythingdefinition.html">Modify a Definition</a></li>
<li><a href="protocol-examples-retrievethingdefinition.html">Retrieve a Definition</a></li>
<li><a href="protocol-examples-createfeatures.html">Create Features</a></li>
<li><a href="protocol-examples-deletefeatures.html">Delete Features</a></li>
<li><a href="protocol-examples-modifyfeatures.html">Modify Features</a></li>
<li><a href="protocol-examples-retrievefeatures.html">Retrieve Features</a></li>
<li><a href="protocol-examples-createfeature.html">Create a single Feature</a></li>
<li><a href="protocol-examples-deletefeature.html">Delete a single Feature</a></li>
<li><a href="protocol-examples-modifyfeature.html">Modify a single Feature</a></li>
<li><a href="protocol-examples-retrievefeature.html">Retrieve a single Feature</a></li>
<li><a href="protocol-examples-createdefinition.html">Create Feature Definition</a></li>
<li><a href="protocol-examples-deletedefinition.html">Delete Feature Definition</a></li>
<li><a href="protocol-examples-modifydefinition.html">Modify Feature Definition</a></li>
<li><a href="protocol-examples-retrievedefinition.html">Retrieve Feature Definition</a></li>
<li><a href="protocol-examples-createproperties.html">Create Feature Properties</a></li>
<li><a href="protocol-examples-deleteproperties.html">Delete Feature Properties</a></li>
<li><a href="protocol-examples-modifyproperties.html">Modify Feature Properties</a></li>
<li><a href="protocol-examples-retrieveproperties.html">Retrieve Feature Properties</a></li>
<li><a href="protocol-examples-createproperty.html">Create a single Property</a></li>
<li><a href="protocol-examples-deleteproperty.html">Delete a single Property</a></li>
<li><a href="protocol-examples-modifyproperty.html">Modify a single Property</a></li>
<li><a href="protocol-examples-retrieveproperty.html">Retrieve a single Property</a></li>
<li><a href="protocol-examples-errorresponses.html">Error responses</a></li>
</ul>
</li>
<li class="subfolders">
<a href="#"><span></span>→ Policies examples</a>
<ul>
<li><a href="protocol-examples-policies-createpolicy.html">Create a Policy</a></li>
<li><a href="protocol-examples-policies-deletepolicy.html">Delete a Policy</a></li>
<li><a href="protocol-examples-policies-modifypolicy.html">Modify a Policy</a></li>
<li><a href="protocol-examples-policies-retrievepolicy.html">Retrieve a Policy</a></li>
<li><a href="protocol-examples-policies-modifypolicyentries.html">Modify entries</a></li>
<li><a href="protocol-examples-policies-retrievepolicyentries.html">Retrieve entries</a></li>
<li><a href="protocol-examples-policies-createpolicyentry.html">Create a single entry</a></li>
<li><a href="protocol-examples-policies-deletepolicyentry.html">Delete a single entry</a></li>
<li><a href="protocol-examples-policies-modifypolicyentry.html">Modify a single entry</a></li>
<li><a href="protocol-examples-policies-retrievepolicyentry.html">Retrieve a single entry</a></li>
<li><a href="protocol-examples-policies-modifysubjects.html">Modify subjects</a></li>
<li><a href="protocol-examples-policies-retrievesubjects.html">Retrieve subjects</a></li>
<li><a href="protocol-examples-policies-createsubject.html">Create a single subject</a></li>
<li><a href="protocol-examples-policies-deletesubject.html">Delete a single subject</a></li>
<li><a href="protocol-examples-policies-modifysubject.html">Modify a single subject</a></li>
<li><a href="protocol-examples-policies-retrievesubject.html">Retrieve a single subject</a></li>
<li><a href="protocol-examples-policies-modifyresources.html">Modify resources</a></li>
<li><a href="protocol-examples-policies-retrieveresources.html">Retrieve resources</a></li>
<li><a href="protocol-examples-policies-createresource.html">Create a single resource</a></li>
<li><a href="protocol-examples-policies-deleteresource.html">Delete a single resource</a></li>
<li><a href="protocol-examples-policies-modifyresource.html">Modify a single resource</a></li>
<li><a href="protocol-examples-policies-retrieveresource.html">Retrieve a single resource</a></li>
<li><a href="protocol-examples-policies-errorresponses.html">Error responses</a></li>
</ul>
</li>
<li><a href="protocol-examples-search.html">→ Search examples</a></li>
</ul>
</li>
<li><a href="sandbox.html">Sandbox</a></li>
<li><a href="presentations.html">Presentations</a></li>
<li><a href="glossary.html">Glossary</a></li>
<li><a href="feedback.html">Feedback</a></li>
<p class="external">
<a href="#" id="collapseAll">Collapse All</a> | <a href="#" id="expandAll">Expand All</a>
</p>
</ul>
<!-- this highlights the active parent class in the sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
<script>$("li.active").parents('li').toggleClass("active");
</script>
</div>
<!-- Content Column -->
<div class="col-md-9" id="tg-sb-content">
<div class="post-header">
<h1 class="post-title-main">Operating Ditto</h1>
</div>
<div class="post-content">
<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
<script>
$( document ).ready(function() {
// Handler for .ready() called.
$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });
/* this offset helps account for the space taken up by the floating toolbar. */
$('#toc').on('click', 'a', function() {
var target = $(this.getAttribute('href'))
, scroll_target = target.offset().top
$(window).scrollTop(scroll_target - 10);
return false
})
});
</script>
<div id="toc"></div>
<p>Once you have successfully started Ditto, proceed with setting it up for continuous operation.</p>
<p>This page shows the basics for operating Ditto.</p>
<h2 id="configuration">Configuration</h2>
<p>Each of Ditto’s microservice has many options for configuration, e.g. timeouts, cache sizes, etc.</p>
<p>In order to have a look at all possible configuration options and what default values they have, here are the configuration
files of Ditto’s microservices:</p>
<ul>
<li>Policies: <a href="https://github.com/eclipse/ditto/blob/master/services/policies/starter/src/main/resources/policies.conf">policies.conf</a></li>
<li>Things: <a href="https://github.com/eclipse/ditto/blob/master/services/things/starter/src/main/resources/things.conf">things.conf</a></li>
<li>Things-Search: <a href="https://github.com/eclipse/ditto/blob/master/services/thingsearch/starter/src/main/resources/things-search.conf">things-search.conf</a></li>
<li>Concierge: <a href="https://github.com/eclipse/ditto/blob/master/services/concierge/starter/src/main/resources/concierge.conf">concierge.conf</a></li>
<li>Connectivity: <a href="https://github.com/eclipse/ditto/blob/master/services/connectivity/starter/src/main/resources/connectivity.conf">connectivity.conf</a></li>
<li>Gateway: <a href="https://github.com/eclipse/ditto/blob/master/services/gateway/starter/src/main/resources/gateway.conf">gateway.conf</a></li>
</ul>
<p>Whenever you find the syntax <code class="highlighter-rouge">${?UPPER_CASE_ENV_NAME}</code> in the configuration files, you may overwrite the default value
by specifying that environment variable when running the container.</p>
<p>When no environment variable is defined in the config, you may change the default value anyway by specifying a
“System property” you pass to the Java process.</p>
<p>The following example configures the devops password of the gateway-service started via docker-compose. In order
to supply additional configuration one has to add the variable in the corresponding <code class="highlighter-rouge">command</code> section of the
<code class="highlighter-rouge">docker-compose.yml</code> file.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="s">...</span>
<span class="s"># Alternative approach for configuration of the service</span>
<span class="s">command</span><span class="pi">:</span> <span class="s">java -Dditto.gateway.authentication.devops.password=foobar -jar starter.jar</span>
</code></pre></div></div>
<p>The executable for the microservice is called <code class="highlighter-rouge">starter.jar</code>. The configuration variables have to be set before
the <code class="highlighter-rouge">-jar</code> option.</p>
<h3 id="pre-authentication">Pre-authentication</h3>
<p>HTTP API calls to Ditto may be authenticated with a reverse proxy (e.g. a nginx) which:</p>
<ul>
<li>authenticates a user/subject</li>
<li>passes the authenticated username as HTTP header</li>
<li>ensures that this HTTP header can never be written by the end-user</li>
</ul>
<p>By default, <code class="highlighter-rouge">pre-authentication</code> is <strong>disabled</strong> in the Ditto <a href="architecture-services-gateway.html">gateway</a> services.<br />
It can however be enabled by configuring the environment variable <code class="highlighter-rouge">ENABLE_PRE_AUTHENTICATION</code> to the value <code class="highlighter-rouge">true</code>.</p>
<p>When it is enabled, the reverse proxy has to set the HTTP header <code class="highlighter-rouge">x-ditto-pre-authenticated</code>.<br />
The format of the “pre-authenticated” string is: <code class="highlighter-rouge">&lt;issuer&gt;:&lt;subject&gt;</code>. The issuer defines which system authenticated the
user and the subject contains e.g. the user-id or -name.</p>
<p>This string must then be used in <a href="basic-policy.html#who-can-be-addressed">policies</a> as “Subject ID”.</p>
<p>Example for a nginx “proxy” configuration:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auth_basic "Authentication required";
auth_basic_user_file nginx.htpasswd;
...
proxy_set_header x-ditto-pre-authenticated "nginx:${remote_user}";
</code></pre></div></div>
<h3 id="openid-connect">OpenID Connect</h3>
<p>The authentication provider must be added to the ditto-gateway configuration.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ditto.gateway.authentication {
oauth {
openid-connect-issuers = {
myprovider = "localhost:9000"
}
}
}
</code></pre></div></div>
<p>In order to do this by specifying a Java system property, use the following:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-Dditto.gateway.authentication.oauth.openid-connect-issuers.myprovider=localhost:9000
</code></pre></div></div>
<p>The configured subject-issuer will be used to prefix the value of the “sub” claim, e.g.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"subjects"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"&lt;provider&gt;:&lt;sub-claim&gt;"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"generated"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>As of the OAuth2.0 and OpenID Connect standards Ditto expects the headers <code class="highlighter-rouge">Authorization: Bearer &lt;JWT&gt;</code> and
<code class="highlighter-rouge">Content-Type: application/json</code>, containing the issued token of the provider.</p>
<p><strong>The token has to be issued beforehand. The required logic is not provided by Ditto.</strong> When using
the OIDC provider <a href="https://www.keycloak.org/">keycloak</a>, a project like <a href="https://github.com/keycloak/keycloak-gatekeeper">keycloak-gatekeeper</a>
may be put in front of Ditto to handle the token-logic.</p>
<p><strong>If the chosen OIDC provider uses a self-signed certificate</strong>, the certificate has to be retrieved and configured for the
akka-http ssl configuration.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssl-config {
trustManager = {
stores = [
{ type = "PEM", path = "/path/to/cert/globalsign.crt" }
]
}
}
</code></pre></div></div>
<h2 id="logging">Logging</h2>
<p>Gathering logs for a running Ditto installation can be achieved by:</p>
<ul>
<li>grepping log output from STDOUT/STDERR via Docker’s <a href="https://docs.docker.com/engine/admin/logging/overview/">logging drivers</a>
<ul>
<li>Benefits: simple, works with all Docker logging drivers (e.g. “awslogs”, “splunk”, etc.)</li>
</ul>
</li>
</ul>
<p>This option may also use an ELK stack with the right Docker logging driver.</p>
<h2 id="monitoring">Monitoring</h2>
<p>In addition to logging, the Ditto images include monitoring features. Specific metrics are
automatically gathered and published on a HTTP port. There it can be scraped by a <a href="https://prometheus.io">Prometheus</a>
backend, from where the metrics can be accessed to display in dashboards (e.g. with <a href="https://grafana.com">Grafana</a>).</p>
<h3 id="configuring">Configuring</h3>
<p>In the default configuration, each Ditto service opens a HTTP endpoint, where it provides the Prometheus metrics on port
<code class="highlighter-rouge">9095</code>. This can be changed via the environment variable <code class="highlighter-rouge">PROMETHEUS_PORT</code>.</p>
<p>Ditto will automatically publish gathered metrics at the endpoint <code class="highlighter-rouge">http://&lt;container-host-or-ip&gt;:9095/</code>.</p>
<p>Further, Prometheus can be configured to poll on all Ditto service endpoints in order to persist the historical metrics.
Grafana can add a Prometheus server as its data source and can display
the metrics based on the keys mentioned in section <a href="#gathered-metrics">“Gathered metrics”</a>.</p>
<h3 id="gathered-metrics">Gathered metrics</h3>
<p>In order to inspect which metrics are exported to Prometheus, just visit the Prometheus HTTP endpoint of a Ditto service:
<code class="highlighter-rouge">http://&lt;container-host-or-ip&gt;:9095/</code>.</p>
<p>The following example shows an excerpt of metrics gathered for the
<a href="architecture-services-gateway.html">gateway-service</a>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#Kamon Metrics
# TYPE jvm_threads gauge
jvm_threads{component="system-metrics",measure="total"} 72.0
# TYPE jvm_memory_buffer_pool_count gauge
jvm_memory_buffer_pool_count{component="system-metrics",pool="direct"} 14.0
# TYPE jvm_class_loading gauge
jvm_class_loading{component="system-metrics",mode="loaded"} 10491.0
# TYPE jvm_memory_buffer_pool_usage gauge
jvm_memory_buffer_pool_usage{component="system-metrics",pool="direct",measure="used"} 396336.0
# TYPE roundtrip_http_seconds histogram
roundtrip_http_seconds_bucket{le="0.05",ditto_request_path="/api/1/things/x",ditto_request_method="PUT",ditto_statusCode="201",segment="overall"} 1.0
roundtrip_http_seconds_sum{ditto_request_path="/api/1/things/x",ditto_statusCode="201",ditto_request_method="PUT",segment="overall"} 0.038273024
roundtrip_http_seconds_bucket{le="0.001",ditto_request_path="/api/1/things/x",ditto_request_method="PUT",ditto_statusCode="204",segment="overall"} 0.0
roundtrip_http_seconds_bucket{le="0.1",ditto_request_path="/api/1/things/x",ditto_request_method="PUT",ditto_statusCode="204",segment="overall"} 7.0
roundtrip_http_seconds_sum{ditto_request_path="/api/1/things/x",ditto_statusCode="204",ditto_request_method="PUT",segment="overall"} 0.828899328
# TYPE jvm_gc_promotion histogram
jvm_gc_promotion_sum{space="old"} 7315456.0
# TYPE jvm_gc_seconds histogram
jvm_gc_seconds_count{component="system-metrics",collector="scavenge"} 9.0
jvm_gc_seconds_sum{component="system-metrics",collector="scavenge"} 0.063
# TYPE jvm_memory_bytes histogram
jvm_memory_bytes_count{component="system-metrics",measure="used",segment="miscellaneous-non-heap-storage"} 54.0
jvm_memory_bytes_sum{component="system-metrics",measure="used",segment="miscellaneous-non-heap-storage"} 786350080.0
</code></pre></div></div>
<p>To put it in a nutshell, Ditto reports:</p>
<ul>
<li>JVM metrics for all services
<ul>
<li>amount of garbage collections + GC times</li>
<li>memory consumption (heap + non-heap)</li>
<li>amount of threads + loaded classes</li>
</ul>
</li>
<li>HTTP metrics for <a href="architecture-services-gateway.html">gateway-service</a>
<ul>
<li>roundtrip times from request to response</li>
<li>amount of HTTP calls</li>
<li>status code of HTTP responses</li>
</ul>
</li>
<li>MongoDB metrics for <a href="architecture-services-things.html">things-service</a>,
<a href="architecture-services-policies.html">policies-service</a>, <a href="architecture-services-things-search.html">things-search-service</a>
<ul>
<li>inserts, updates, reads per second</li>
<li>roundtrip times</li>
</ul>
</li>
<li>cache metrics for <a href="architecture-services-concierge.html">concierge-service</a></li>
<li>connection metrics for <a href="architecture-services-connectivity.html">connectivity-service</a>
<ul>
<li>processed messages</li>
<li>mapping times</li>
</ul>
</li>
</ul>
<h2 id="devops-commands">DevOps commands</h2>
<p>The “DevOps commands” API allows Ditto operators to make changes to a running installation without restarts.</p>
<p>The following DevOps commands are supported:</p>
<ul>
<li>Dynamically retrieve and change log levels</li>
<li>Dynamically retrieve service configuration</li>
<li>Piggyback commands</li>
</ul>
<div class="alert alert-info" role="alert"><i class="fa fa-info-circle"></i> <b>Note:</b> The default credentials for the <code class="highlighter-rouge">/devops</code> HTTP endpoint are username: <code class="highlighter-rouge">devops</code>, password: <code class="highlighter-rouge">foobar</code>. The password can be changed by setting the environment variable <code class="highlighter-rouge">DEVOPS_PASSWORD</code> in the gateway service.</div>
<h3 id="dynamically-adjust-log-levels">Dynamically adjust log levels</h3>
<p>Changing the log levels dynamically is very useful when debugging an accidental problem,
since the cause of the problem could be lost on service restart.</p>
<h4 id="retrieve-all-log-levels">Retrieve all log levels</h4>
<p>Example for retrieving all currently configured log levels:<br />
<code class="highlighter-rouge">GET /devops/logging</code></p>
<p>Response:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"gateway"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"devops.responses:retrieveLoggerConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"serviceName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"gateway"</span><span class="p">,</span><span class="w">
</span><span class="s2">"instance"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="s2">"loggerConfigs"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"info"</span><span class="p">,</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ROOT"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"info"</span><span class="p">,</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.eclipse.ditto"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"warn"</span><span class="p">,</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.mongodb.driver"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"things-search"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"policies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"things"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"connectivity"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h4 id="change-a-specific-log-level-for-all-services">Change a specific log level for all services</h4>
<p>Example request payload to change the log level of logger <code class="highlighter-rouge">org.eclipse.ditto</code> in all services to <code class="highlighter-rouge">DEBUG</code>:<br />
<code class="highlighter-rouge">PUT /devops/logging</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.eclipse.ditto"</span><span class="p">,</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"debug"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h4 id="retrieve-log-levels-of-a-service">Retrieve log levels of a service</h4>
<p>Example response for retrieving all currently configured log levels of gateways services:<br />
<code class="highlighter-rouge">GET /devops/logging/gateway</code></p>
<p>Response:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"devops.responses:retrieveLoggerConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"serviceName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"gateway"</span><span class="p">,</span><span class="w">
</span><span class="s2">"instance"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="s2">"loggerConfigs"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"info"</span><span class="p">,</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ROOT"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"info"</span><span class="p">,</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.eclipse.ditto"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"warn"</span><span class="p">,</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.mongodb.driver"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h4 id="change-a-specific-log-level-for-one-service">Change a specific log level for one service</h4>
<p>Example request payload to change the log level of logger <code class="highlighter-rouge">org.eclipse.ditto</code> in all
instances of gateway-service to <code class="highlighter-rouge">DEBUG</code>:</p>
<p><code class="highlighter-rouge">PUT /devops/logging/gateway</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"logger"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.eclipse.ditto"</span><span class="p">,</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="s2">"debug"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h3 id="dynamically-retrieve-configurations">Dynamically retrieve configurations</h3>
<p>Runtime configurations of services are available for the Ditto operator at
<code class="highlighter-rouge">/devops/config/</code> with optional restrictions by service name, instance ID and configuration path.
The entire runtime configuration of a service may be dozens of kilobytes big. If it exceeds the cluster message size
of 250 kB, then it can only be read piece by piece via the <code class="highlighter-rouge">path</code> query parameter.</p>
<h4 id="retrieve-all-service-configurations">Retrieve all service configurations</h4>
<p>Retrieve the configuration at the path <code class="highlighter-rouge">ditto.info</code> thus:</p>
<p><code class="highlighter-rouge">GET /devops/config?path=ditto.info</code></p>
<p>It is recommended to not omit the query parameter <code class="highlighter-rouge">path</code>. Otherwise the full configurations of all services are
aggregated in the response, which can become megabytes big.</p>
<p>The path <code class="highlighter-rouge">ditto.info</code> points to information on service name, service instance index, JVM arguments and environment
variables. Response example:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.responses:retrieveConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"config"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"env"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"PATH"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/usr/games:/usr/local/games"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"service"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"instance-index"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w">
</span><span class="s2">"service-name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"gateway"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"vm-args"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"-Dfile.encoding=UTF-8"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"?1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.responses:retrieveConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"config"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"env"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"CONNECTIVITY_FLUSH_PENDING_RESPONSES_TIMEOUT"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3d"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"service"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"instance-index"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w">
</span><span class="s2">"service-name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"connectivity"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"vm-args"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"-Dditto.connectivity.connection.snapshot.threshold=2"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h4 id="retrieve-the-configuration-of-a-service-instance">Retrieve the configuration of a service instance</h4>
<p>Retrieving the configuration of a specific service instance is much faster
because the response is not aggregated from an unknown number of respondents
over the duration given in the query parameter <code class="highlighter-rouge">timeout</code>.</p>
<p>To retrieve <code class="highlighter-rouge">ditto</code> configuration from Gateway instance <code class="highlighter-rouge">1</code>:</p>
<p><code class="highlighter-rouge">GET /devops/config/gateway/1?path=ditto</code></p>
<p>Response example:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.responses:retrieveConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"config"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"cluster"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"number-of-shards"</span><span class="p">:</span><span class="w"> </span><span class="mi">20</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"gateway"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"authentication"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"devops"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"password"</span><span class="p">:</span><span class="w"> </span><span class="s2">"foobar"</span><span class="p">,</span><span class="w">
</span><span class="s2">"securestatus"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h3 id="piggyback-commands">Piggyback commands</h3>
<p>You can use a DevOps command to send a command to another actor in the cluster.
Those special commands are called piggyback commands.
A piggyback command must conform to the following schema:</p>
<script src="docson/widget.js" data-schema="../jsonschema/piggyback-command.json"></script>
<p>Example:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/system/sharding/connection"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"connectivity.commands:createConnection"</span><span class="p">,</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h4 id="managing-connections">Managing connections</h4>
<p>Piggybacks are used to configure Dittos connectivity service. More information on this can be found in
the <a href="connectivity-manage-connections.html">Manage Connections</a> section.</p>
<h4 id="managing-background-cleanup">Managing background cleanup</h4>
<p>Ditto deletes unnecessary events and snapshots in the background according to database load.
<a href="architecture-services-concierge.html">Concierge</a> has a cluster-singleton coordinating the background cleanup process.
The cluster singleton responds to piggyback-commands to query its state and configuration, modify the configuration,
and restart the background cleanup process.</p>
<p>Each command is sent to the actor selection <code class="highlighter-rouge">/user/conciergeRoot/eventSnapshotCleanupCoordinatorProxy</code> on <em>one</em>
Concierge instance, typically <code class="highlighter-rouge">INSTANCE_INDEX=1</code> in a docker-based installation:</p>
<p><code class="highlighter-rouge">POST /devops/piggygack/concierge/&lt;INSTANCE_INDEX&gt;?timeout=10s</code></p>
<h5 id="query-background-cleanup-coordinator-state">Query background cleanup coordinator state</h5>
<p><code class="highlighter-rouge">POST /devops/piggygack/concierge/&lt;INSTANCE_INDEX&gt;?timeout=10s</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/user/conciergeRoot/eventSnapshotCleanupCoordinatorProxy"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="s2">"is-grouped-topic"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"status.commands:retrieveHealth"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>The response has the following details:</p>
<ul>
<li><code class="highlighter-rouge">events</code>: State transitions of the actor. The top entry is the current state of the actor.</li>
<li><code class="highlighter-rouge">credit-decisions</code>: Decisions on how many cleanup actions were permitted, when, and why.</li>
<li><code class="highlighter-rouge">actions</code>: Log of cleanup actions, their round-trip times, and whether they were successful.</li>
</ul>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"status.responses:retrieveHealth"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"statusInfo"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"UP"</span><span class="p">,</span><span class="w">
</span><span class="s2">"details"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"INFO"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"events"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"2019-06-24T13:42:29.878Z"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Stream terminated. Result=&lt;Done&gt; Error=&lt;null&gt;"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"2019-06-24T13:42:19.474Z"</span><span class="p">:</span><span class="w"> </span><span class="s2">"WOKE_UP"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"credit-decisions"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"2019-06-24T13:42:29.609Z"</span><span class="p">:</span><span class="w"> </span><span class="s2">"100: maxTimeNanos=0 is below threshold=20000000"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"2019-06-24T13:42:25.232Z"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0: maxTimeNanos=47358000 is above threshold=20000000"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"actions"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"2019-06-24T13:42:28.801Z"</span><span class="p">:</span><span class="w"> </span><span class="s2">"200 start=2019-06-24T13:42:28.755Z &lt;thing:ditto:thing1&gt;"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h5 id="query-background-cleanup-coordinator-configuration">Query background cleanup coordinator configuration</h5>
<p><code class="highlighter-rouge">POST /devops/piggygack/concierge/&lt;INSTANCE_INDEX&gt;?timeout=10s</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/user/conciergeRoot/eventSnapshotCleanupCoordinatorProxy"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="s2">"is-grouped-topic"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.commands:retrieveConfig"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Response example:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.responses:retrieveConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"config"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"cleanup-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"30s"</span><span class="p">,</span><span class="w">
</span><span class="s2">"credit-decision"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"credit-per-batch"</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="p">,</span><span class="w">
</span><span class="s2">"interval"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10s"</span><span class="p">,</span><span class="w">
</span><span class="s2">"metric-report-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10s"</span><span class="p">,</span><span class="w">
</span><span class="s2">"timer-threshold"</span><span class="p">:</span><span class="w"> </span><span class="s2">"20ms"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"keep"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"actions"</span><span class="p">:</span><span class="w"> </span><span class="mi">120</span><span class="p">,</span><span class="w">
</span><span class="s2">"credit-decisions"</span><span class="p">:</span><span class="w"> </span><span class="mi">30</span><span class="p">,</span><span class="w">
</span><span class="s2">"events"</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"parallelism"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="s2">"persistence-ids"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"burst"</span><span class="p">:</span><span class="w"> </span><span class="mi">25</span><span class="p">,</span><span class="w">
</span><span class="s2">"stream-idle-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10m"</span><span class="p">,</span><span class="w">
</span><span class="s2">"stream-request-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10s"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"quiet-period"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5m"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h5 id="modify-background-cleanup-coordinator-configuration">Modify background cleanup coordinator configuration</h5>
<p>Send a piggyback command of type <code class="highlighter-rouge">common.commands:modifyConfig</code> to change the configuration of the background cleanup
coordinator. All subsequent cleanup processes will use the new configuration. Any ongoing cleanup is not affected.
Configurations absent in the payload of the piggyback command remain unchanged.</p>
<p><code class="highlighter-rouge">POST /devops/piggygack/concierge/&lt;INSTANCE_INDEX&gt;?timeout=10s</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/user/conciergeRoot/eventSnapshotCleanupCoordinatorProxy"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="s2">"is-grouped-topic"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.commands:modifyConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"config"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"quiet-period"</span><span class="p">:</span><span class="w"> </span><span class="s2">"240d"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>The response contains the effective configuration of the background cleanup coordinator. If the configuration in the
piggyback command contains any error, then an error is logged and the actor’s configuration is unchanged.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.responses:modifyConfig"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"config"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"cleanup-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"30s"</span><span class="p">,</span><span class="w">
</span><span class="s2">"credit-decision"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"credit-per-batch"</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="p">,</span><span class="w">
</span><span class="s2">"interval"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10s"</span><span class="p">,</span><span class="w">
</span><span class="s2">"metric-report-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10s"</span><span class="p">,</span><span class="w">
</span><span class="s2">"timer-threshold"</span><span class="p">:</span><span class="w"> </span><span class="s2">"20ms"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"keep"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"actions"</span><span class="p">:</span><span class="w"> </span><span class="mi">120</span><span class="p">,</span><span class="w">
</span><span class="s2">"credit-decisions"</span><span class="p">:</span><span class="w"> </span><span class="mi">30</span><span class="p">,</span><span class="w">
</span><span class="s2">"events"</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"parallelism"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="s2">"persistence-ids"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"burst"</span><span class="p">:</span><span class="w"> </span><span class="mi">25</span><span class="p">,</span><span class="w">
</span><span class="s2">"stream-idle-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10m"</span><span class="p">,</span><span class="w">
</span><span class="s2">"stream-request-timeout"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10s"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"quiet-period"</span><span class="p">:</span><span class="w"> </span><span class="s2">"240d"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h5 id="shutdown-background-cleanup-coordinator">Shutdown background cleanup coordinator</h5>
<p>Send a piggyback command of type <code class="highlighter-rouge">common.commands:shutdown</code> to stop the background cleanup process.
The next process is scheduled after the <code class="highlighter-rouge">quiet-period</code> duration in the coordinator’s configuration.</p>
<p><code class="highlighter-rouge">POST /devops/piggygack/concierge/&lt;INSTANCE_INDEX&gt;?timeout=10s</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/user/conciergeRoot/eventSnapshotCleanupCoordinatorProxy"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="s2">"is-grouped-topic"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.commands:shutdown"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Response example:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"common.responses:shutdown"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Restarting stream in &lt;PT5760H30M5S&gt;."</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h5 id="cleanup-events-and-snapshots-of-an-entity">Cleanup events and snapshots of an entity</h5>
<p>Send a cleanup command by piggyback to the entity’s service and shard region to trigger removal of stale events and
snapshots manually. Here is an example for things. Change the service name and shard region name accordingly for
policies and connections. Typically in a docker based environment, use <code class="highlighter-rouge">INSTANCE_INDEX=1</code>.</p>
<p><code class="highlighter-rouge">POST /devops/piggygack/things/&lt;INSTANCE_INDEX&gt;?timeout=10s</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/system/sharding/thing"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cleanup.commands:cleanupPersistence"</span><span class="p">,</span><span class="w">
</span><span class="s2">"entityId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ditto:thing1"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Response example:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"?"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cleanup.responses:cleanupPersistence"</span><span class="p">,</span><span class="w">
</span><span class="s2">"status"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="s2">"entityId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"thing:ditto:thing1"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h4 id="managing-background-synchronization">Managing background synchronization</h4>
<p>A background sync actor goes over thing snapshots and search index entries slowly to ensure eventual consistency
of the search index. The actor operates in the same manner as the background cleanup coordinator and responds to
the same commands.</p>
<p><code class="highlighter-rouge">POST /devops/piggygack/things-search/&lt;INSTANCE_INDEX&gt;?timeout=10s</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/user/thingsSearchRoot/searchUpdaterRoot/backgroundSyncProxy"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="s2">"is-grouped-topic"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;COMMAND-TYPE&gt;"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p><code class="highlighter-rouge">COMMAND-TYPE</code> can be:</p>
<ul>
<li><code class="highlighter-rouge">common.commands:shutdown</code> to shutdown or restart a background sync stream,</li>
<li><code class="highlighter-rouge">common.commands:retrieveConfig</code> to retrieve the current configuration,</li>
<li><code class="highlighter-rouge">common.commands:modifyConfig</code> to modify the current configuration, or</li>
<li><code class="highlighter-rouge">status.commands:retrieveHealth</code> to query the current progress and event log.</li>
</ul>
<p>For each command type, please refer to the corresponding segment of “Managing background cleanup” for the exact format.</p>
<h4 id="force-search-index-update-for-one-thing">Force search index update for one thing</h4>
<p>The search index should rarely become out-of-sync for a long time, and it can repair itself
of any inconsistencies detected at query time. Nevertheless, you can trigger search index update
for a particular thing by a DevOp-command and bring the entry up-to-date immediately.</p>
<p><code class="highlighter-rouge">POST /devops/piggygack/things-search/&lt;INSTANCE_INDEX&gt;?timeout=0</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/user/thingsSearchRoot/searchUpdaterRoot/thingsUpdater"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"aggregate"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="s2">"is-grouped-topic"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"piggybackCommand"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"thing-search.commands:updateThing"</span><span class="p">,</span><span class="w">
</span><span class="s2">"thingId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&lt;THING-ID&gt;"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>There is no response. Things-search service will log a warning upon receiving this message
and continue to log warnings should the search index update fail on the persistence.</p>
<h4 id="erasing-data-within-a-namespace">Erasing data within a namespace</h4>
<p>Ditto supports erasure of <em>all</em> data within a namespace during live operations.
To do so safely, perform the following steps in sequence.</p>
<ol>
<li><a href="#block-all-messages-to-a-namespace">Block all messages to the namespace</a>
so that actors will not spawn in the namespace.</li>
<li><a href="#shutdown-all-actors-in-a-namespace">Shutdown all actors in the namespace</a>
so that no actor will generate data in the namespace.</li>
<li><a href="#erase-all-data-in-a-namespace-from-the-persistence">Erase data from the persistence</a>.</li>
<li><a href="#unblock-messages-to-a-namespace">Unblock messages to the namespace</a>
so that the old namespace could be reused at a later point in time.</li>
</ol>
<h5 id="block-all-messages-to-a-namespace">Block all messages to a namespace</h5>
<p>Send a piggyback command to <a href="https://doc.akka.io/docs/akka/current/distributed-pub-sub.html">Akka’s pub-sub-mediator</a> with type <code class="highlighter-rouge">namespaces.commands:blockNamespace</code>
to block all messages sent to actors belonging to a namespace.</p>
<p><code class="highlighter-rouge">PUT /devops/piggygack?timeout=10s</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"targetActorSelection"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/system/distributedPubSubMediator"</span><span class="p">,</span><span class="w">
</span><span class="s2">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">